Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:54.950992 27851 test_util.cc:268] Using random seed: 1296859408
2024-05-03T14:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:11:54Z Disabled control of system clock
I20240503 14:11:54.988209 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.254:36571
--webserver_interface=127.27.50.254
--webserver_port=0
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240503 14:11:55.454985 27859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:55.455543 27859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:55.455749 27859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:55.466962 27859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:55.467152 27859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:55.467250 27859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:55.467381 27859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:55.479300 27859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.254:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.50.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:55.480947 27859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:55.483250 27859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:55.497637 27867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.497583 27865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.497702 27864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.499759 27859 server_base.cc:1034] running on GCE node
I20240503 14:11:55.500777 27859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:55.502521 27859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:55.503849 27859 hybrid_clock.cc:648] HybridClock initialized: now 1714745515503706 us; error 70 us; skew 500 ppm
I20240503 14:11:55.504369 27859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.512545 27859 webserver.cc:460] Webserver started at http://127.27.50.254:38435/ using document root <none> and password file <none>
I20240503 14:11:55.513489 27859 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.513664 27859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.514039 27859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.517587 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/instance:
uuid: "bb6a7aee99214df4b02858ddae025b14"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-8wrt"
I20240503 14:11:55.518558 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal/instance:
uuid: "bb6a7aee99214df4b02858ddae025b14"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-8wrt"
I20240503 14:11:55.525873 27859 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240503 14:11:55.530544 27873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.532435 27859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240503 14:11:55.532692 27859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
uuid: "bb6a7aee99214df4b02858ddae025b14"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-8wrt"
I20240503 14:11:55.532974 27859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.551261 27859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.552433 27859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:55.552911 27859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.582139 27859 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.254:36571
I20240503 14:11:55.582191 27924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.254:36571 every 8 connection(s)
I20240503 14:11:55.583946 27859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
I20240503 14:11:55.589130 27925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:55.590801 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 27859
I20240503 14:11:55.591123 27851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal/instance
I20240503 14:11:55.594059 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.253:44259
--webserver_interface=127.27.50.253
--webserver_port=0
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 14:11:55.596936 27925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:55.617594 27925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:11:55.618852 27874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111)
W20240503 14:11:55.620735 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.253:44259: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111)
I20240503 14:11:55.670181 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } attempt: 1
W20240503 14:11:55.672390 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.253:44259: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111)
I20240503 14:11:55.740725 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } attempt: 2
W20240503 14:11:55.742748 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.253:44259: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111)
I20240503 14:11:55.834080 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } attempt: 3
W20240503 14:11:55.836246 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.253:44259: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111)
I20240503 14:11:55.979566 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } attempt: 4
W20240503 14:11:55.981801 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.253:44259: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111)
W20240503 14:11:56.051527 27927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.051890 27927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.052073 27927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.061947 27927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:56.062122 27927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.062227 27927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:56.062345 27927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:56.073603 27927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.253:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.50.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.075152 27927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.077262 27927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.090298 27935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.090341 27938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.090421 27936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.091445 27927 server_base.cc:1034] running on GCE node
I20240503 14:11:56.092350 27927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.094839 27927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.096136 27927 hybrid_clock.cc:648] HybridClock initialized: now 1714745516095983 us; error 75 us; skew 500 ppm
I20240503 14:11:56.096527 27927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.099556 27927 webserver.cc:460] Webserver started at http://127.27.50.253:43185/ using document root <none> and password file <none>
I20240503 14:11:56.100486 27927 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.100649 27927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.100978 27927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.104153 27927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/instance:
uuid: "a10ddca38c26462eb221d6026e88c6cc"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-8wrt"
I20240503 14:11:56.105031 27927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal/instance:
uuid: "a10ddca38c26462eb221d6026e88c6cc"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-8wrt"
I20240503 14:11:56.111781 27927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240503 14:11:56.116389 27944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.118094 27927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:11:56.118322 27927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
uuid: "a10ddca38c26462eb221d6026e88c6cc"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-8wrt"
I20240503 14:11:56.118544 27927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.148262 27927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.149361 27927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.149744 27927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.178203 27927 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.253:44259
I20240503 14:11:56.178251 27995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.253:44259 every 8 connection(s)
I20240503 14:11:56.179927 27927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/info.pb
I20240503 14:11:56.184779 27996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:56.184770 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 27927
I20240503 14:11:56.184974 27851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal/instance
I20240503 14:11:56.186177 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.252:45363
--webserver_interface=127.27.50.252
--webserver_port=0
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 14:11:56.192112 27996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:56.209255 27996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:56.220551 27996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:11:56.221947 27946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
W20240503 14:11:56.223985 27996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
I20240503 14:11:56.273386 27996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 1
W20240503 14:11:56.275590 27996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
I20240503 14:11:56.281126 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } attempt: 5
I20240503 14:11:56.288578 27925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:11:56.290897 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
I20240503 14:11:56.342255 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 1
I20240503 14:11:56.343878 27996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 2
W20240503 14:11:56.344709 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
W20240503 14:11:56.346199 27996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
I20240503 14:11:56.413069 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 2
W20240503 14:11:56.415831 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
I20240503 14:11:56.437556 27996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 3
W20240503 14:11:56.440213 27996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
I20240503 14:11:56.522166 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 3
W20240503 14:11:56.524675 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
I20240503 14:11:56.583503 27996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 4
W20240503 14:11:56.585935 27996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
W20240503 14:11:56.665570 27998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.665982 27998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.666179 27998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.675721 27998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:56.675880 27998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.675979 27998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:56.676127 27998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:56.687774 27998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.252:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.50.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.689275 27998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.691272 27998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:11:56.701980 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 4
W20240503 14:11:56.704463 27925 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.50.252:45363: Network error: Client connection negotiation failed: client connection to 127.27.50.252:45363: connect: Connection refused (error 111)
W20240503 14:11:56.705049 28007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.705143 28008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.705219 28010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.706876 27998 server_base.cc:1034] running on GCE node
I20240503 14:11:56.707899 27998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.709782 27998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.711118 27998 hybrid_clock.cc:648] HybridClock initialized: now 1714745516710982 us; error 68 us; skew 500 ppm
I20240503 14:11:56.711577 27998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.714852 27998 webserver.cc:460] Webserver started at http://127.27.50.252:34653/ using document root <none> and password file <none>
I20240503 14:11:56.715869 27998 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.716114 27998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.716521 27998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.719954 27998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data/instance:
uuid: "c0690a516c89446d85c88cdf64fa4238"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-8wrt"
I20240503 14:11:56.721088 27998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/wal/instance:
uuid: "c0690a516c89446d85c88cdf64fa4238"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-8wrt"
I20240503 14:11:56.728402 27998 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240503 14:11:56.733175 28016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.735102 27998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:11:56.735354 27998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/wal
uuid: "c0690a516c89446d85c88cdf64fa4238"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-8wrt"
I20240503 14:11:56.735607 27998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.771848 27998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.772946 27998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.773317 27998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.801021 27998 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.252:45363
I20240503 14:11:56.801081 28067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.252:45363 every 8 connection(s)
I20240503 14:11:56.802724 27998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/data/info.pb
I20240503 14:11:56.807384 28068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:56.812397 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 27998
I20240503 14:11:56.812638 27851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-2/wal/instance
I20240503 14:11:56.814615 28068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:56.831549 28068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:56.839689 28068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:56.857908 28068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Bootstrap starting.
I20240503 14:11:56.862107 28068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:56.863853 28068 log.cc:824] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:56.866967 28068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: No bootstrap required, opened a new log
I20240503 14:11:56.874992 28068 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:56.875393 28068 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:56.875581 28068 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0690a516c89446d85c88cdf64fa4238, State: Initialized, Role: FOLLOWER
I20240503 14:11:56.876315 28068 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:56.877530 28079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:56.877959 28079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:56.878630 28068 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:56.885273 27996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 5
I20240503 14:11:56.895697 28079 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:56.896095 28079 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:56.898488 28079 leader_election.cc:288] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259)
I20240503 14:11:56.899508 27996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Bootstrap starting.
I20240503 14:11:56.904731 27996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:56.889446 27998 master_runner.cc:186] Error getting master registration for 127.27.50.254:36571: OK, instance_id { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" instance_seqno: 1714745515574161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 14:11:56.906062 27998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 14:11:56.906601 27996 log.cc:824] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Log is configured to *not* fsync() on all Append() calls
W20240503 14:11:56.909485 28090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:11:56.909718 28090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:11:56.911001 27996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: No bootstrap required, opened a new log
I20240503 14:11:56.922489 27996 raft_consensus.cc:348] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:56.923060 27996 raft_consensus.cc:374] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:56.923259 27996 raft_consensus.cc:729] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10ddca38c26462eb221d6026e88c6cc, State: Initialized, Role: FOLLOWER
I20240503 14:11:56.924271 27996 consensus_queue.cc:260] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:56.925840 28093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:56.926486 28093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:56.927443 27996 sys_catalog.cc:564] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:56.929509 27971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0690a516c89446d85c88cdf64fa4238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10ddca38c26462eb221d6026e88c6cc" is_pre_election: true
I20240503 14:11:56.930311 27971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0690a516c89446d85c88cdf64fa4238 in term 0.
I20240503 14:11:56.931176 28017 leader_election.cc:302] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc, c0690a516c89446d85c88cdf64fa4238; no voters:
I20240503 14:11:56.931792 28079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:56.931998 28079 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:56.932183 28079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:56.936324 28079 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:56.938848 28079 leader_election.cc:288] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [CANDIDATE]: Term 1 election: Requested vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259)
I20240503 14:11:56.938984 27971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0690a516c89446d85c88cdf64fa4238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10ddca38c26462eb221d6026e88c6cc"
I20240503 14:11:56.939306 27971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:56.943961 27971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0690a516c89446d85c88cdf64fa4238 in term 1.
I20240503 14:11:56.945796 28017 leader_election.cc:302] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc, c0690a516c89446d85c88cdf64fa4238; no voters:
I20240503 14:11:56.946364 28079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:56.947614 28079 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [term 1 LEADER]: Becoming Leader. State: Replica: c0690a516c89446d85c88cdf64fa4238, State: Running, Role: LEADER
I20240503 14:11:56.948876 28079 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:56.962109 28102 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0690a516c89446d85c88cdf64fa4238. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:56.962688 28102 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:56.973584 28106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:56.967868 27927 master_runner.cc:186] Error getting master registration for 127.27.50.254:36571: OK, instance_id { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" instance_seqno: 1714745515574161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 14:11:56.980693 28106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:11:56.980751 27927 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240503 14:11:56.981227 28107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:11:56.981371 28107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:11:56.981735 27925 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } attempt: 5
I20240503 14:11:57.003705 27971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Refusing update from remote peer c0690a516c89446d85c88cdf64fa4238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:11:57.004818 27925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Bootstrap starting.
I20240503 14:11:57.005084 28102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:57.016463 27925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:57.021528 27925 log.cc:824] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Log is configured to *not* fsync() on all Append() calls
W20240503 14:11:57.026171 28022 tablet.cc:2369] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:11:57.038669 28093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0690a516c89446d85c88cdf64fa4238. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:57.039108 28093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:57.041747 27925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: No bootstrap required, opened a new log
I20240503 14:11:57.045954 28079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:57.046384 28079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:57.046494 28110 mvcc.cc:205] Tried to move back new op lower bound from 7023597637633409024 to 7023597637434908672. Current Snapshot: MvccSnapshot[applied={T|T < 7023597637633409024}]
I20240503 14:11:57.049979 28106 catalog_manager.cc:1347] Generated new cluster ID: 09434ad4cf0745ecb5d037e3cd83df07
I20240503 14:11:57.050163 28106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:57.063768 28093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:57.064035 28093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:57.063278 27925 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:57.064273 27925 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:57.064610 27925 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6a7aee99214df4b02858ddae025b14, State: Initialized, Role: FOLLOWER
I20240503 14:11:57.066130 27925 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:57.079885 27900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0690a516c89446d85c88cdf64fa4238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6a7aee99214df4b02858ddae025b14" is_pre_election: true
I20240503 14:11:57.080413 27898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:57.084275 27899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0690a516c89446d85c88cdf64fa4238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6a7aee99214df4b02858ddae025b14"
I20240503 14:11:57.088634 27898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Refusing update from remote peer c0690a516c89446d85c88cdf64fa4238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:11:57.089314 28106 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:57.090255 28102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:57.091356 28106 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:57.096899 27925 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:57.111069 28106 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Generated new TSK 0
I20240503 14:11:57.112434 28106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:57.130744 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.50.193:0
--local_ip_for_outbound_sockets=127.27.50.193
--webserver_interface=127.27.50.193
--webserver_port=0
--tserver_master_addrs=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 14:11:57.157261 28115 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:57.157465 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0690a516c89446d85c88cdf64fa4238. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:57.157893 28115 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:57.158021 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:57.158304 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:57.158705 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:57.160723 28079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:11:57.161105 28079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:57.239068 28134 catalog_manager.cc:1259] Loaded cluster ID: 09434ad4cf0745ecb5d037e3cd83df07
I20240503 14:11:57.239236 28134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: loading cluster ID for follower catalog manager: success
I20240503 14:11:57.259753 28134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: acquiring CA information for follower catalog manager: success
I20240503 14:11:57.263080 28134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240503 14:11:57.899719 28127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.900450 28127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.900732 28127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.916069 28127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.916512 28127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.50.193
I20240503 14:11:57.935046 28127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.50.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.50.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.50.193
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.937234 28127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.940337 28127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.965368 28143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.965271 28141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.966279 28140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.967767 28127 server_base.cc:1034] running on GCE node
I20240503 14:11:57.968851 28127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.970445 28127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.971805 28127 hybrid_clock.cc:648] HybridClock initialized: now 1714745517971684 us; error 53 us; skew 500 ppm
I20240503 14:11:57.972371 28127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.976696 28127 webserver.cc:460] Webserver started at http://127.27.50.193:44891/ using document root <none> and password file <none>
I20240503 14:11:57.978044 28127 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.978314 28127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.978767 28127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.983424 28127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data/instance:
uuid: "a64bc15d65d84ad5a005960a9a84df9c"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-8wrt"
I20240503 14:11:57.983980 28107 catalog_manager.cc:1259] Loaded cluster ID: 09434ad4cf0745ecb5d037e3cd83df07
I20240503 14:11:57.984160 28107 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: loading cluster ID for follower catalog manager: success
I20240503 14:11:57.984865 28127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/instance:
uuid: "a64bc15d65d84ad5a005960a9a84df9c"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-8wrt"
I20240503 14:11:57.987776 28107 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: acquiring CA information for follower catalog manager: success
I20240503 14:11:57.990447 28107 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:11:57.993546 28127 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20240503 14:11:57.998688 28149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.000838 28127 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240503 14:11:58.001091 28127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal
uuid: "a64bc15d65d84ad5a005960a9a84df9c"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-8wrt"
I20240503 14:11:58.001345 28127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.031738 28127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.032992 28127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.033468 28127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.034945 28127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.037472 28127 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.037595 28127 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.037730 28127 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.037801 28127 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.092110 28127 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.193:44355
I20240503 14:11:58.092190 28261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.193:44355 every 8 connection(s)
I20240503 14:11:58.093843 28127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/data/info.pb
I20240503 14:11:58.095279 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 28127
I20240503 14:11:58.095571 27851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/instance
I20240503 14:11:58.115931 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.50.194:0
--local_ip_for_outbound_sockets=127.27.50.194
--webserver_interface=127.27.50.194
--webserver_port=0
--tserver_master_addrs=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 14:11:58.128931 28273 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:11:58.129526 28273 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:58.132704 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6a7aee99214df4b02858ddae025b14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a10ddca38c26462eb221d6026e88c6cc" is_pre_election: true
I20240503 14:11:58.134371 28273 leader_election.cc:288] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:11:58.135351 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6a7aee99214df4b02858ddae025b14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:11:58.136965 27875 leader_election.cc:302] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb6a7aee99214df4b02858ddae025b14; no voters: a10ddca38c26462eb221d6026e88c6cc, c0690a516c89446d85c88cdf64fa4238
I20240503 14:11:58.138386 28273 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:11:58.145026 28262 heartbeater.cc:347] Connected to a master server at 127.27.50.252:45363
I20240503 14:11:58.145551 28262 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.145761 28264 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:11:58.145972 28264 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.147122 28264 heartbeater.cc:511] Master 127.27.50.253:44259 requested a full tablet report, sending...
I20240503 14:11:58.147186 28262 heartbeater.cc:511] Master 127.27.50.252:45363 requested a full tablet report, sending...
I20240503 14:11:58.150190 27961 ts_manager.cc:194] Registered new tserver with Master: a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:11:58.150219 28033 ts_manager.cc:194] Registered new tserver with Master: a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:11:58.150728 28263 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:11:58.150956 28263 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.151839 28263 heartbeater.cc:511] Master 127.27.50.254:36571 requested a full tablet report, sending...
I20240503 14:11:58.152863 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.50.193:45997
I20240503 14:11:58.154881 27890 ts_manager.cc:194] Registered new tserver with Master: a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:11:58.412760 28274 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:11:58.413012 28274 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:58.414659 28274 leader_election.cc:288] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:11:58.415510 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:11:58.415510 27899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bb6a7aee99214df4b02858ddae025b14" is_pre_election: true
I20240503 14:11:58.416563 27946 leader_election.cc:302] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc; no voters: bb6a7aee99214df4b02858ddae025b14, c0690a516c89446d85c88cdf64fa4238
I20240503 14:11:58.417136 28274 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:11:58.611950 28272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:58.612417 28272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:58.612622 28272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:58.622238 28272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:58.622538 28272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.50.194
I20240503 14:11:58.634346 28272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.50.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.50.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.50.194
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:58.635968 28272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:58.638146 28272 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.654192 28282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.654664 28280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.654573 28279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.656798 28262 heartbeater.cc:503] Master 127.27.50.252:45363 was elected leader, sending a full tablet report...
I20240503 14:11:58.657128 28272 server_base.cc:1034] running on GCE node
I20240503 14:11:58.658036 28272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:58.659544 28272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:58.660825 28272 hybrid_clock.cc:648] HybridClock initialized: now 1714745518660707 us; error 63 us; skew 500 ppm
I20240503 14:11:58.661237 28272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.664379 28272 webserver.cc:460] Webserver started at http://127.27.50.194:41841/ using document root <none> and password file <none>
I20240503 14:11:58.665308 28272 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.665485 28272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.665853 28272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.669322 28272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data/instance:
uuid: "d3a6b9b7913444d29c96581af6febf33"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-8wrt"
I20240503 14:11:58.670315 28272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/instance:
uuid: "d3a6b9b7913444d29c96581af6febf33"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-8wrt"
I20240503 14:11:58.677953 28272 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20240503 14:11:58.680598 28288 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:11:58.680824 28288 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:11:58.681712 28288 leader_election.cc:288] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:11:58.682442 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6a7aee99214df4b02858ddae025b14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:11:58.682449 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6a7aee99214df4b02858ddae025b14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a10ddca38c26462eb221d6026e88c6cc" is_pre_election: true
I20240503 14:11:58.683260 27874 leader_election.cc:302] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb6a7aee99214df4b02858ddae025b14; no voters: a10ddca38c26462eb221d6026e88c6cc, c0690a516c89446d85c88cdf64fa4238
I20240503 14:11:58.683511 28289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.683609 28288 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:11:58.685504 28272 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20240503 14:11:58.685842 28272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal
uuid: "d3a6b9b7913444d29c96581af6febf33"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-8wrt"
I20240503 14:11:58.686123 28272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.714496 28272 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.715633 28272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.716034 28272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.717437 28272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.719683 28272 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.719797 28272 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.719929 28272 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.719991 28272 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.770548 28272 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.194:33463
I20240503 14:11:58.770601 28401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.194:33463 every 8 connection(s)
I20240503 14:11:58.772217 28272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/data/info.pb
I20240503 14:11:58.772615 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 28272
I20240503 14:11:58.772862 27851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/instance
I20240503 14:11:58.783021 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.50.195:0
--local_ip_for_outbound_sockets=127.27.50.195
--webserver_interface=127.27.50.195
--webserver_port=0
--tserver_master_addrs=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240503 14:11:58.817641 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:11:58.817626 28403 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:11:58.818163 28406 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.818171 28402 heartbeater.cc:347] Connected to a master server at 127.27.50.252:45363
I20240503 14:11:58.818346 28402 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.818385 28403 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.820174 28402 heartbeater.cc:511] Master 127.27.50.252:45363 requested a full tablet report, sending...
I20240503 14:11:58.820222 28406 heartbeater.cc:511] Master 127.27.50.253:44259 requested a full tablet report, sending...
I20240503 14:11:58.820607 28403 heartbeater.cc:511] Master 127.27.50.254:36571 requested a full tablet report, sending...
I20240503 14:11:58.822265 28033 ts_manager.cc:194] Registered new tserver with Master: d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:11:58.822415 27890 ts_manager.cc:194] Registered new tserver with Master: d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:11:58.822552 27961 ts_manager.cc:194] Registered new tserver with Master: d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:11:58.823450 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.50.194:43807
W20240503 14:11:59.289033 28407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:59.289368 28407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:59.289520 28407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:59.299280 28407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:59.299667 28407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.50.195
I20240503 14:11:59.311035 28407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.50.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.50.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.50.195
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:59.312546 28407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:59.314540 28407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:11:59.326349 28402 heartbeater.cc:503] Master 127.27.50.252:45363 was elected leader, sending a full tablet report...
W20240503 14:11:59.330054 28418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:59.330227 28420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:59.330456 28417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:59.332235 28407 server_base.cc:1034] running on GCE node
I20240503 14:11:59.332919 28407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:59.334051 28407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:59.335306 28407 hybrid_clock.cc:648] HybridClock initialized: now 1714745519335202 us; error 54 us; skew 500 ppm
I20240503 14:11:59.335667 28407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:59.338620 28407 webserver.cc:460] Webserver started at http://127.27.50.195:33289/ using document root <none> and password file <none>
I20240503 14:11:59.339515 28407 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:59.339690 28407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:59.340058 28407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:59.343263 28407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data/instance:
uuid: "d472d9f94d8949cf86284d32a161cebc"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-8wrt"
I20240503 14:11:59.344219 28407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/instance:
uuid: "d472d9f94d8949cf86284d32a161cebc"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-8wrt"
I20240503 14:11:59.350797 28407 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20240503 14:11:59.355724 28426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.357509 28407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240503 14:11:59.357748 28407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal
uuid: "d472d9f94d8949cf86284d32a161cebc"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-8wrt"
I20240503 14:11:59.357990 28407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:59.379012 28407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:59.380210 28407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:59.380674 28407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:59.382272 28407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:59.384724 28407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:59.384842 28407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.384996 28407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:59.385092 28407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.471710 28407 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.195:35233
I20240503 14:11:59.471781 28538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.195:35233 every 8 connection(s)
I20240503 14:11:59.473453 28407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/data/info.pb
I20240503 14:11:59.481189 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 28407
I20240503 14:11:59.481611 27851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/instance
I20240503 14:11:59.522541 28540 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:11:59.523049 28540 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:59.524757 28540 heartbeater.cc:511] Master 127.27.50.254:36571 requested a full tablet report, sending...
I20240503 14:11:59.527463 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:11:59.527722 28541 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:59.528236 27890 ts_manager.cc:194] Registered new tserver with Master: d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:11:59.528538 28541 heartbeater.cc:511] Master 127.27.50.253:44259 requested a full tablet report, sending...
I20240503 14:11:59.530472 27961 ts_manager.cc:194] Registered new tserver with Master: d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:11:59.531581 28539 heartbeater.cc:347] Connected to a master server at 127.27.50.252:45363
I20240503 14:11:59.531749 28539 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:59.532565 28539 heartbeater.cc:511] Master 127.27.50.252:45363 requested a full tablet report, sending...
I20240503 14:11:59.533936 28033 ts_manager.cc:194] Registered new tserver with Master: d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:11:59.535311 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.50.195:55307
I20240503 14:11:59.547399 27851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 14:11:59.587464 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3b6b7883895e4de2955baa6a2d00f7ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:11:59.588604 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_43c52adefa824fcb9c57c2d78385807f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.589689 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b6b7883895e4de2955baa6a2d00f7ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:11:59.589689 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43c52adefa824fcb9c57c2d78385807f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.598153 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3ca2b27559e84275ad25910d7effa8e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.599225 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ca2b27559e84275ad25910d7effa8e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.607124 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0a89e43de47c4ee2ba100e7dbf451d84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.609844 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a89e43de47c4ee2ba100e7dbf451d84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.611629 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_68cb064eabe94eabad42adf809501dd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.612663 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68cb064eabe94eabad42adf809501dd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.621855 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b4cbc679204745a8aac3fa9095567939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.622862 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4cbc679204745a8aac3fa9095567939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.634953 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a230793d86bf4bf7a1420e821c7b8997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.637455 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a230793d86bf4bf7a1420e821c7b8997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.647823 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_bc04ce776cfb4771977f33fa236ea77f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.648733 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc04ce776cfb4771977f33fa236ea77f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.661440 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_40ba6c787988402c8a38fd3a451d5fe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.662271 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40ba6c787988402c8a38fd3a451d5fe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.670850 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2be03633b8854e03955f07503a5309a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.671676 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be03633b8854e03955f07503a5309a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.675138 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_43c52adefa824fcb9c57c2d78385807f" } new_table_name: "default.Table_20759dec9179422e982b8a26c8572788" modify_external_catalogs: true
I20240503 14:11:59.679154 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3b6b7883895e4de2955baa6a2d00f7ea" } new_table_name: "default.Table_c75bd5158ad74ccd9cc5ffd60749907a" modify_external_catalogs: true
I20240503 14:11:59.685091 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b4cbc679204745a8aac3fa9095567939" } new_table_name: "default.Table_4a16c898f66d4ad999025524253f572b" modify_external_catalogs: true
I20240503 14:11:59.718834 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_68cb064eabe94eabad42adf809501dd4" } new_table_name: "default.Table_b7c3bfbf0a1c4619b9da6d0a65e1f65d" modify_external_catalogs: true
I20240503 14:11:59.726128 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3ca2b27559e84275ad25910d7effa8e9" } new_table_name: "default.Table_45cb217a6f5745f0a3042bae42e41088" modify_external_catalogs: true
I20240503 14:11:59.731495 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0a89e43de47c4ee2ba100e7dbf451d84" } modify_external_catalogs: true
I20240503 14:11:59.731868 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0a89e43de47c4ee2ba100e7dbf451d84" } modify_external_catalogs: true
W20240503 14:11:59.737957 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d37413e13809448ea8c94ae0e9efc618 has no consensus state
W20240503 14:11:59.739192 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e20b99c022ea437181f8d3221b952932 consensus state has no leader
W20240503 14:11:59.740311 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bfc56c4591c437da5b9686b726686b2 has no consensus state
I20240503 14:11:59.753147 28474 tablet_service.cc:1467] Processing CreateTablet for tablet e20b99c022ea437181f8d3221b952932 (DEFAULT_TABLE table=default.Table_3b6b7883895e4de2955baa6a2d00f7ea [id=d2b505b742e64fa6948bf41c834d8902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.760016 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20b99c022ea437181f8d3221b952932. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:11:59.770794 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet d37413e13809448ea8c94ae0e9efc618 (table default.Table_4a16c898f66d4ad999025524253f572b [id=ebd0741cd2a143698fd3cea6cf80b853], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d37413e13809448ea8c94ae0e9efc618 has no consensus state
I20240503 14:11:59.816946 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c75bd5158ad74ccd9cc5ffd60749907a" } modify_external_catalogs: true
I20240503 14:11:59.817343 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c75bd5158ad74ccd9cc5ffd60749907a" } modify_external_catalogs: true
I20240503 14:11:59.818287 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_bc04ce776cfb4771977f33fa236ea77f" } modify_external_catalogs: true
I20240503 14:11:59.818667 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_bc04ce776cfb4771977f33fa236ea77f" } modify_external_catalogs: true
W20240503 14:11:59.822249 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3826594d300342c7a59e0eee989a873f has no consensus state
I20240503 14:11:59.830261 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da (DEFAULT_TABLE table=default.Table_bc04ce776cfb4771977f33fa236ea77f [id=a9b7085b3a83482a91e6a8c888956a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.835568 28472 tablet_service.cc:1467] Processing CreateTablet for tablet fdccb370198b4eb69768aeed787fd682 (DEFAULT_TABLE table=default.Table_a230793d86bf4bf7a1420e821c7b8997 [id=9f10df997dc04f05a4f052f20c4a2fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.835456 28197 tablet_service.cc:1467] Processing CreateTablet for tablet e20b99c022ea437181f8d3221b952932 (DEFAULT_TABLE table=default.Table_3b6b7883895e4de2955baa6a2d00f7ea [id=d2b505b742e64fa6948bf41c834d8902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.836614 28193 tablet_service.cc:1467] Processing CreateTablet for tablet fd98387c1cae45dcb29c2116b6141708 (DEFAULT_TABLE table=default.Table_3ca2b27559e84275ad25910d7effa8e9 [id=bd615ce41c41475fa37c3d6de18ece3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.836823 28470 tablet_service.cc:1467] Processing CreateTablet for tablet fd98387c1cae45dcb29c2116b6141708 (DEFAULT_TABLE table=default.Table_3ca2b27559e84275ad25910d7effa8e9 [id=bd615ce41c41475fa37c3d6de18ece3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.837153 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da (DEFAULT_TABLE table=default.Table_bc04ce776cfb4771977f33fa236ea77f [id=a9b7085b3a83482a91e6a8c888956a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.837695 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd98387c1cae45dcb29c2116b6141708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.837921 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 3826594d300342c7a59e0eee989a873f (DEFAULT_TABLE table=default.Table_b7c3bfbf0a1c4619b9da6d0a65e1f65d [id=208684b1dfd044c6af3b7b4fa83b8c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.838199 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ee72bcd4654d06a91ccfb7c24869da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.838766 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfc56c4591c437da5b9686b726686b2 (DEFAULT_TABLE table=default.Table_20759dec9179422e982b8a26c8572788 [id=3a24797cc53b439da44cc77025fa9f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.839743 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdccb370198b4eb69768aeed787fd682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.839653 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ee72bcd4654d06a91ccfb7c24869da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.840870 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfc56c4591c437da5b9686b726686b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.841229 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd98387c1cae45dcb29c2116b6141708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.841153 28195 tablet_service.cc:1467] Processing CreateTablet for tablet fdccb370198b4eb69768aeed787fd682 (DEFAULT_TABLE table=default.Table_a230793d86bf4bf7a1420e821c7b8997 [id=9f10df997dc04f05a4f052f20c4a2fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.842211 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdccb370198b4eb69768aeed787fd682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.842517 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20b99c022ea437181f8d3221b952932. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.843384 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 42d0a58592c946bfa57a5698add39985 (DEFAULT_TABLE table=default.Table_0a89e43de47c4ee2ba100e7dbf451d84 [id=72babfde5906464b9a4e55e85c5db68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.843585 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3826594d300342c7a59e0eee989a873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.845140 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d0a58592c946bfa57a5698add39985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.850153 28337 tablet_service.cc:1467] Processing CreateTablet for tablet e20b99c022ea437181f8d3221b952932 (DEFAULT_TABLE table=default.Table_3b6b7883895e4de2955baa6a2d00f7ea [id=d2b505b742e64fa6948bf41c834d8902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.851881 28335 tablet_service.cc:1467] Processing CreateTablet for tablet fdccb370198b4eb69768aeed787fd682 (DEFAULT_TABLE table=default.Table_a230793d86bf4bf7a1420e821c7b8997 [id=9f10df997dc04f05a4f052f20c4a2fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.857326 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdccb370198b4eb69768aeed787fd682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.857560 28330 tablet_service.cc:1467] Processing CreateTablet for tablet d37413e13809448ea8c94ae0e9efc618 (DEFAULT_TABLE table=default.Table_4a16c898f66d4ad999025524253f572b [id=ebd0741cd2a143698fd3cea6cf80b853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.857563 28190 tablet_service.cc:1467] Processing CreateTablet for tablet d37413e13809448ea8c94ae0e9efc618 (DEFAULT_TABLE table=default.Table_4a16c898f66d4ad999025524253f572b [id=ebd0741cd2a143698fd3cea6cf80b853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.858417 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37413e13809448ea8c94ae0e9efc618. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:11:59.859265 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd98387c1cae45dcb29c2116b6141708 consensus state has no leader
I20240503 14:11:59.852864 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfc56c4591c437da5b9686b726686b2 (DEFAULT_TABLE table=default.Table_20759dec9179422e982b8a26c8572788 [id=3a24797cc53b439da44cc77025fa9f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.859737 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_40ba6c787988402c8a38fd3a451d5fe0" } modify_external_catalogs: true
I20240503 14:11:59.853281 28333 tablet_service.cc:1467] Processing CreateTablet for tablet fd98387c1cae45dcb29c2116b6141708 (DEFAULT_TABLE table=default.Table_3ca2b27559e84275ad25910d7effa8e9 [id=bd615ce41c41475fa37c3d6de18ece3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.859974 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfc56c4591c437da5b9686b726686b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.860561 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37413e13809448ea8c94ae0e9efc618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.860569 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd98387c1cae45dcb29c2116b6141708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.853710 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 3826594d300342c7a59e0eee989a873f (DEFAULT_TABLE table=default.Table_b7c3bfbf0a1c4619b9da6d0a65e1f65d [id=208684b1dfd044c6af3b7b4fa83b8c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.861562 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2088aba8de5d4a309d04f4f5d77d43bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:11:59.861876 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 42d0a58592c946bfa57a5698add39985 (DEFAULT_TABLE table=default.Table_0a89e43de47c4ee2ba100e7dbf451d84 [id=72babfde5906464b9a4e55e85c5db68e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:11:59.862376 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2088aba8de5d4a309d04f4f5d77d43bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.862625 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d0a58592c946bfa57a5698add39985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.867204 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfc56c4591c437da5b9686b726686b2 (DEFAULT_TABLE table=default.Table_20759dec9179422e982b8a26c8572788 [id=3a24797cc53b439da44cc77025fa9f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.868011 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfc56c4591c437da5b9686b726686b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.872465 28467 tablet_service.cc:1467] Processing CreateTablet for tablet d37413e13809448ea8c94ae0e9efc618 (DEFAULT_TABLE table=default.Table_4a16c898f66d4ad999025524253f572b [id=ebd0741cd2a143698fd3cea6cf80b853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.873355 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37413e13809448ea8c94ae0e9efc618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.859635 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_449d6f39397e46358bc986a8c98a3ea9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:11:59.875970 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_40ba6c787988402c8a38fd3a451d5fe0" } modify_external_catalogs: true
I20240503 14:11:59.854594 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 42d0a58592c946bfa57a5698add39985 (DEFAULT_TABLE table=default.Table_0a89e43de47c4ee2ba100e7dbf451d84 [id=72babfde5906464b9a4e55e85c5db68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.855188 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da (DEFAULT_TABLE table=default.Table_bc04ce776cfb4771977f33fa236ea77f [id=a9b7085b3a83482a91e6a8c888956a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.878021 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 3826594d300342c7a59e0eee989a873f (DEFAULT_TABLE table=default.Table_b7c3bfbf0a1c4619b9da6d0a65e1f65d [id=208684b1dfd044c6af3b7b4fa83b8c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:59.881728 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3826594d300342c7a59e0eee989a873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.888567 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3826594d300342c7a59e0eee989a873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.889392 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ee72bcd4654d06a91ccfb7c24869da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.891413 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d0a58592c946bfa57a5698add39985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.893362 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20b99c022ea437181f8d3221b952932. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:11:59.899349 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_449d6f39397e46358bc986a8c98a3ea9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.902300 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_71af49729f324730af4113012443a75c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.903182 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71af49729f324730af4113012443a75c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.904234 28600 tablet_bootstrap.cc:492] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:11:59.910372 28600 tablet_bootstrap.cc:654] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.911098 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 90ee72bcd4654d06a91ccfb7c24869da
I20240503 14:11:59.912281 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 42d0a58592c946bfa57a5698add39985
I20240503 14:11:59.912778 28600 log.cc:824] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:59.914885 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d0a58592c946bfa57a5698add39985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:11:59.915028 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d0a58592c946bfa57a5698add39985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:11:59.907707 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_666d072a7d2a4affa01a3cf64abf777c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.943667 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_666d072a7d2a4affa01a3cf64abf777c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.945425 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d0a58592c946bfa57a5698add39985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:11:59.954612 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:11:59.963300 28600 tablet_bootstrap.cc:492] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:11:59.963860 28600 ts_tablet_manager.cc:1367] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.016s
I20240503 14:11:59.916106 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4e83fe8e46644b8f8df871f6da4f0f1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.970000 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 42d0a58592c946bfa57a5698add39985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d0a58592c946bfa57a5698add39985 already in progress: creating tablet
I20240503 14:11:59.973270 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0515e83b3d914915a67f1d30f123eb09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:11:59.974169 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:11:59.974978 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2ec7eed485554ccbb4bd34b53bdc7618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.975833 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:11:59.977634 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e83fe8e46644b8f8df871f6da4f0f1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.978040 28600 raft_consensus.cc:348] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:11:59.978796 28600 raft_consensus.cc:374] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.979001 28600 raft_consensus.cc:729] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
W20240503 14:11:59.980355 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ec7eed485554ccbb4bd34b53bdc7618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.980404 28600 consensus_queue.cc:260] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:11:59.983923 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 42d0a58592c946bfa57a5698add39985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d0a58592c946bfa57a5698add39985 already in progress: creating tablet
W20240503 14:11:59.984795 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 90ee72bcd4654d06a91ccfb7c24869da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90ee72bcd4654d06a91ccfb7c24869da already in progress: creating tablet
I20240503 14:11:59.986719 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:11:59.987008 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:11:59.989023 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 90ee72bcd4654d06a91ccfb7c24869da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90ee72bcd4654d06a91ccfb7c24869da already in progress: creating tablet
W20240503 14:11:59.989401 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0515e83b3d914915a67f1d30f123eb09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.000320 28030 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 6914d1ec5cae4c679d9a20ea34dacff1
I20240503 14:12:00.000950 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a9c4dd16181e4b0789779c3f25eaed1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.002210 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9c4dd16181e4b0789779c3f25eaed1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.003979 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e20b99c022ea437181f8d3221b952932
I20240503 14:12:00.006112 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet e20b99c022ea437181f8d3221b952932 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.006109 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_35f27fac8e5342d28c1883f89c3491c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.006417 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet e20b99c022ea437181f8d3221b952932 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.006841 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet e20b99c022ea437181f8d3221b952932 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.007339 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_45cb217a6f5745f0a3042bae42e41088" } new_table_name: "default.Table_059ac6056dc04f58b62976bc4cb523a2" modify_external_catalogs: true
W20240503 14:12:00.008368 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e20b99c022ea437181f8d3221b952932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20b99c022ea437181f8d3221b952932 already in progress: creating tablet
W20240503 14:12:00.008802 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e20b99c022ea437181f8d3221b952932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20b99c022ea437181f8d3221b952932 already in progress: creating tablet
W20240503 14:12:00.009328 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35f27fac8e5342d28c1883f89c3491c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.010773 28600 ts_tablet_manager.cc:1398] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.047s user 0.015s sys 0.010s
I20240503 14:12:00.029145 28600 tablet_bootstrap.cc:492] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.033847 28600 tablet_bootstrap.cc:654] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.038561 28539 heartbeater.cc:503] Master 127.27.50.252:45363 was elected leader, sending a full tablet report...
I20240503 14:12:00.042656 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a230793d86bf4bf7a1420e821c7b8997" } new_table_name: "default.Table_9c60ef813e134942b2bf7a67b5f6ecf4" modify_external_catalogs: true
I20240503 14:12:00.051023 28600 tablet_bootstrap.cc:492] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.051245 28600 ts_tablet_manager.cc:1367] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.005s
I20240503 14:12:00.052587 28600 raft_consensus.cc:348] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.052886 28600 raft_consensus.cc:374] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.053030 28600 raft_consensus.cc:729] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.053310 28600 consensus_queue.cc:260] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.053388 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b7c3bfbf0a1c4619b9da6d0a65e1f65d" } new_table_name: "default.Table_ad66d05a032c41f0b4e2ec45e9044b31" modify_external_catalogs: true
I20240503 14:12:00.057006 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2be03633b8854e03955f07503a5309a5" } modify_external_catalogs: true
I20240503 14:12:00.057307 28600 ts_tablet_manager.cc:1398] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:00.057338 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2be03633b8854e03955f07503a5309a5" } modify_external_catalogs: true
I20240503 14:12:00.062227 28600 tablet_bootstrap.cc:492] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.067111 28600 tablet_bootstrap.cc:654] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.074097 28600 tablet_bootstrap.cc:492] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.074384 28600 ts_tablet_manager.cc:1367] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240503 14:12:00.075479 28600 raft_consensus.cc:348] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.075809 28600 raft_consensus.cc:374] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.075896 28600 raft_consensus.cc:729] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.076326 28600 consensus_queue.cc:260] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.077483 28614 tablet_bootstrap.cc:492] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.083212 28600 ts_tablet_manager.cc:1398] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:12:00.090246 28600 tablet_bootstrap.cc:492] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.091107 28614 tablet_bootstrap.cc:654] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.093389 28600 tablet_bootstrap.cc:654] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.095436 28614 log.cc:824] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:00.109874 28614 tablet_bootstrap.cc:492] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.112027 28614 ts_tablet_manager.cc:1367] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.000s
I20240503 14:12:00.115761 28600 tablet_bootstrap.cc:492] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.116008 28600 ts_tablet_manager.cc:1367] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.012s
I20240503 14:12:00.117185 28600 raft_consensus.cc:348] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.117532 28600 raft_consensus.cc:374] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.117671 28600 raft_consensus.cc:729] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.118161 28600 consensus_queue.cc:260] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:00.125510 28265 tablet.cc:2369] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:00.129123 28600 ts_tablet_manager.cc:1398] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 14:12:00.133793 28604 tablet_bootstrap.cc:492] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.135993 28600 tablet_bootstrap.cc:492] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.139341 28600 tablet_bootstrap.cc:654] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.143932 28614 raft_consensus.cc:348] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.144668 28614 raft_consensus.cc:374] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.144840 28614 raft_consensus.cc:729] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.145829 28614 consensus_queue.cc:260] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.146062 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_20759dec9179422e982b8a26c8572788" } new_table_name: "default.Table_5cbe683faac947458db5ca9867db6c58" modify_external_catalogs: true
I20240503 14:12:00.150498 28614 ts_tablet_manager.cc:1398] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.037s user 0.013s sys 0.003s
I20240503 14:12:00.151683 28600 tablet_bootstrap.cc:492] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.152115 28600 ts_tablet_manager.cc:1367] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240503 14:12:00.153298 28600 raft_consensus.cc:348] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:00.153658 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 42d0a58592c946bfa57a5698add39985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d0a58592c946bfa57a5698add39985 already in progress: creating tablet
I20240503 14:12:00.153713 28600 raft_consensus.cc:374] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.153934 28600 raft_consensus.cc:729] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.153995 28614 tablet_bootstrap.cc:492] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.154397 28600 consensus_queue.cc:260] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.155169 28600 ts_tablet_manager.cc:1398] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:00.155707 28600 tablet_bootstrap.cc:492] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.156329 28614 tablet_bootstrap.cc:654] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.157590 28604 tablet_bootstrap.cc:654] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.159716 28600 tablet_bootstrap.cc:654] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.170502 28604 log.cc:824] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:00.171033 28600 tablet_bootstrap.cc:492] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.171308 28600 ts_tablet_manager.cc:1367] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
W20240503 14:12:00.171653 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 90ee72bcd4654d06a91ccfb7c24869da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90ee72bcd4654d06a91ccfb7c24869da already in progress: creating tablet
W20240503 14:12:00.172073 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e20b99c022ea437181f8d3221b952932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20b99c022ea437181f8d3221b952932 already in progress: creating tablet
I20240503 14:12:00.174764 28614 tablet_bootstrap.cc:492] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.175861 28614 ts_tablet_manager.cc:1367] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20240503 14:12:00.177774 28614 raft_consensus.cc:348] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.178308 28614 raft_consensus.cc:374] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.178617 28614 raft_consensus.cc:729] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.179226 28614 consensus_queue.cc:260] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.180250 28614 ts_tablet_manager.cc:1398] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:00.181308 28614 tablet_bootstrap.cc:492] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.184612 28614 tablet_bootstrap.cc:654] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.194063 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_449d6f39397e46358bc986a8c98a3ea9" } new_table_name: "default.Table_25378c7373494eb6a7e7e3bebd175e6b" modify_external_catalogs: true
I20240503 14:12:00.196291 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet d37413e13809448ea8c94ae0e9efc618 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.200114 28614 tablet_bootstrap.cc:492] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.201828 28614 ts_tablet_manager.cc:1367] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20240503 14:12:00.203222 28614 raft_consensus.cc:348] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.205699 28614 raft_consensus.cc:374] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.206233 28614 raft_consensus.cc:729] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.206642 28600 raft_consensus.cc:348] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.207093 28600 raft_consensus.cc:374] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.207281 28600 raft_consensus.cc:729] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.207826 28600 consensus_queue.cc:260] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.208669 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 (DEFAULT_TABLE table=default.Table_2be03633b8854e03955f07503a5309a5 [id=7907cfab39b74beba4ceddd1762bffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.209252 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71598d0a44384a3b93dec30d6c60ecd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.212862 28600 ts_tablet_manager.cc:1398] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.041s user 0.004s sys 0.000s
I20240503 14:12:00.214242 28600 tablet_bootstrap.cc:492] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.215464 28614 consensus_queue.cc:260] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.219640 28614 ts_tablet_manager.cc:1398] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20240503 14:12:00.220976 28614 tablet_bootstrap.cc:492] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.224166 28614 tablet_bootstrap.cc:654] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.225317 28600 tablet_bootstrap.cc:654] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.227710 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 (DEFAULT_TABLE table=default.Table_2be03633b8854e03955f07503a5309a5 [id=7907cfab39b74beba4ceddd1762bffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.228396 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71598d0a44384a3b93dec30d6c60ecd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.229235 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 (DEFAULT_TABLE table=default.Table_2be03633b8854e03955f07503a5309a5 [id=7907cfab39b74beba4ceddd1762bffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.229774 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71598d0a44384a3b93dec30d6c60ecd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.229763 28600 tablet_bootstrap.cc:492] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.230031 28600 ts_tablet_manager.cc:1367] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240503 14:12:00.231210 28600 raft_consensus.cc:348] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.231527 28600 raft_consensus.cc:374] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.231624 28600 raft_consensus.cc:729] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.232033 28600 consensus_queue.cc:260] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.239924 28600 ts_tablet_manager.cc:1398] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:12:00.242635 28600 tablet_bootstrap.cc:492] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.243009 28604 tablet_bootstrap.cc:492] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:00.245173 28600 tablet_bootstrap.cc:654] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.246852 28604 ts_tablet_manager.cc:1367] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.122s user 0.020s sys 0.010s
W20240503 14:12:00.257257 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d637087c60144aaaa9b0965110d280e has no consensus state
I20240503 14:12:00.257711 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet d37413e13809448ea8c94ae0e9efc618 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.261801 28614 tablet_bootstrap.cc:492] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.262220 28614 ts_tablet_manager.cc:1367] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.004s
I20240503 14:12:00.263353 28614 raft_consensus.cc:348] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.263752 28614 raft_consensus.cc:374] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.263972 28614 raft_consensus.cc:729] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.264142 27851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu pid 27859
I20240503 14:12:00.264386 28600 tablet_bootstrap.cc:492] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.264621 28614 consensus_queue.cc:260] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.264631 28600 ts_tablet_manager.cc:1367] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240503 14:12:00.267073 28614 ts_tablet_manager.cc:1398] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:12:00.267033 28600 raft_consensus.cc:348] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.267331 28600 raft_consensus.cc:374] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.267534 28600 raft_consensus.cc:729] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.267907 28614 tablet_bootstrap.cc:492] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.271246 27920 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.271340 28614 tablet_bootstrap.cc:654] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.272339 28604 raft_consensus.cc:348] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.272974 28604 raft_consensus.cc:374] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.273123 28604 raft_consensus.cc:729] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.274137 28604 consensus_queue.cc:260] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.275051 28600 consensus_queue.cc:260] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.275931 28600 ts_tablet_manager.cc:1398] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:12:00.276410 28600 tablet_bootstrap.cc:492] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.278831 28600 tablet_bootstrap.cc:654] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.282199 28604 ts_tablet_manager.cc:1398] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.035s user 0.010s sys 0.007s
I20240503 14:12:00.282868 28604 tablet_bootstrap.cc:492] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.285836 28604 tablet_bootstrap.cc:654] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.286944 28031 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet d37413e13809448ea8c94ae0e9efc618 (table default.Table_4a16c898f66d4ad999025524253f572b [id=ebd0741cd2a143698fd3cea6cf80b853]). Expected version 1 got 0
I20240503 14:12:00.287331 28031 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_20759dec9179422e982b8a26c8572788 [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 1 got 0
I20240503 14:12:00.287629 28031 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3826594d300342c7a59e0eee989a873f (table default.Table_b7c3bfbf0a1c4619b9da6d0a65e1f65d [id=208684b1dfd044c6af3b7b4fa83b8c2b]). Expected version 1 got 0
I20240503 14:12:00.287882 28031 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet fd98387c1cae45dcb29c2116b6141708 (table default.Table_45cb217a6f5745f0a3042bae42e41088 [id=bd615ce41c41475fa37c3d6de18ece3a]). Expected version 1 got 0
W20240503 14:12:00.289335 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3826594d300342c7a59e0eee989a873f consensus state has no leader
I20240503 14:12:00.289490 28614 tablet_bootstrap.cc:492] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.289741 28614 ts_tablet_manager.cc:1367] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.007s
I20240503 14:12:00.289757 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet d37413e13809448ea8c94ae0e9efc618 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.289851 28600 tablet_bootstrap.cc:492] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.290076 28600 ts_tablet_manager.cc:1367] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20240503 14:12:00.290843 28614 raft_consensus.cc:348] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.291217 28614 raft_consensus.cc:374] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.291142 28600 raft_consensus.cc:348] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.291334 28614 raft_consensus.cc:729] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.291349 28604 tablet_bootstrap.cc:492] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:00.291415 28600 raft_consensus.cc:374] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.291522 28600 raft_consensus.cc:729] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.291535 28604 ts_tablet_manager.cc:1367] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
W20240503 14:12:00.291750 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bfc56c4591c437da5b9686b726686b2 consensus state has no leader
I20240503 14:12:00.291739 28614 consensus_queue.cc:260] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.291895 28600 consensus_queue.cc:260] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:00.292371 28408 tablet.cc:2369] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:00.292333 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e" } modify_external_catalogs: true
I20240503 14:12:00.292588 28600 ts_tablet_manager.cc:1398] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:00.292604 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e" } modify_external_catalogs: true
W20240503 14:12:00.294870 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdccb370198b4eb69768aeed787fd682 consensus state has no leader
I20240503 14:12:00.294951 28604 raft_consensus.cc:348] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.295244 28604 raft_consensus.cc:374] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.295312 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet fd98387c1cae45dcb29c2116b6141708 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.295280 28637 raft_consensus.cc:482] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.295411 28604 raft_consensus.cc:729] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.295743 28604 consensus_queue.cc:260] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.295703 28637 raft_consensus.cc:504] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.296427 28604 ts_tablet_manager.cc:1398] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240503 14:12:00.296919 28614 ts_tablet_manager.cc:1398] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 14:12:00.297421 28614 tablet_bootstrap.cc:492] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.299751 28614 tablet_bootstrap.cc:654] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.304986 28604 tablet_bootstrap.cc:492] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.306627 28637 leader_election.cc:288] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.307674 28604 tablet_bootstrap.cc:654] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.312247 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6782582fb543f4b621569343995dfc (DEFAULT_TABLE table=default.Table_35f27fac8e5342d28c1883f89c3491c1 [id=9f62dbc5baa94df18b25548d731ddeee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.312916 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6782582fb543f4b621569343995dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.321799 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6782582fb543f4b621569343995dfc (DEFAULT_TABLE table=default.Table_35f27fac8e5342d28c1883f89c3491c1 [id=9f62dbc5baa94df18b25548d731ddeee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.322449 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6782582fb543f4b621569343995dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.323115 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6782582fb543f4b621569343995dfc (DEFAULT_TABLE table=default.Table_35f27fac8e5342d28c1883f89c3491c1 [id=9f62dbc5baa94df18b25548d731ddeee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.323643 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6782582fb543f4b621569343995dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.332437 28600 tablet_bootstrap.cc:492] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.333710 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 67ec68f75fa2435698383a03874a5bae (DEFAULT_TABLE table=default.Table_a9c4dd16181e4b0789779c3f25eaed1c [id=18925bb0895246988de55da2f108454b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.334436 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ec68f75fa2435698383a03874a5bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.346915 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 67ec68f75fa2435698383a03874a5bae (DEFAULT_TABLE table=default.Table_a9c4dd16181e4b0789779c3f25eaed1c [id=18925bb0895246988de55da2f108454b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.347096 28473 tablet_service.cc:1467] Processing CreateTablet for tablet f35e87db1c2a42a79b426b2e7767367a (DEFAULT_TABLE table=default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.347738 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35e87db1c2a42a79b426b2e7767367a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.348028 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ec68f75fa2435698383a03874a5bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.354008 28330 tablet_service.cc:1467] Processing CreateTablet for tablet f35e87db1c2a42a79b426b2e7767367a (DEFAULT_TABLE table=default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.354594 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35e87db1c2a42a79b426b2e7767367a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.357224 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b (DEFAULT_TABLE table=default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e [id=e6b422e744c14453a7e7c4b28c1eaac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.357760 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c0611cf7aa4ff3b4e7e6081558729b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.365736 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 (DEFAULT_TABLE table=default.Table_4e83fe8e46644b8f8df871f6da4f0f1d [id=a1a30b8a5adb44b390e6bea2dd82c20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.365752 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b (DEFAULT_TABLE table=default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e [id=e6b422e744c14453a7e7c4b28c1eaac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.366298 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa994dfa54a47408bfb34bcfda9a415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.366734 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c0611cf7aa4ff3b4e7e6081558729b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.374495 28614 tablet_bootstrap.cc:492] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.374626 28604 tablet_bootstrap.cc:492] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:00.374884 28604 ts_tablet_manager.cc:1367] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.070s user 0.014s sys 0.034s
I20240503 14:12:00.374876 28614 ts_tablet_manager.cc:1367] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.078s user 0.024s sys 0.040s
I20240503 14:12:00.375882 28614 raft_consensus.cc:348] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.376173 28614 raft_consensus.cc:374] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.376322 28614 raft_consensus.cc:729] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.376614 28614 consensus_queue.cc:260] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.377447 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd98387c1cae45dcb29c2116b6141708" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:00.353420 28193 tablet_service.cc:1467] Processing CreateTablet for tablet f35e87db1c2a42a79b426b2e7767367a (DEFAULT_TABLE table=default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.378809 28614 ts_tablet_manager.cc:1398] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:00.379055 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35e87db1c2a42a79b426b2e7767367a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.379348 28614 tablet_bootstrap.cc:492] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.379753 28600 tablet_bootstrap.cc:654] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.365275 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b (DEFAULT_TABLE table=default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e [id=e6b422e744c14453a7e7c4b28c1eaac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.373198 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 (DEFAULT_TABLE table=default.Table_4e83fe8e46644b8f8df871f6da4f0f1d [id=a1a30b8a5adb44b390e6bea2dd82c20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.381912 28614 tablet_bootstrap.cc:654] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.382930 28604 raft_consensus.cc:348] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.383303 28604 raft_consensus.cc:374] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.383510 28604 raft_consensus.cc:729] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.383859 28604 consensus_queue.cc:260] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.384649 28604 ts_tablet_manager.cc:1398] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:12:00.389484 28329 tablet_service.cc:1467] Processing CreateTablet for tablet f54dc481079c4a5db5b54d485991fa65 (DEFAULT_TABLE table=default.Table_2088aba8de5d4a309d04f4f5d77d43bd [id=3e4bfd54a342491b86e978c08faa5c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.390192 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54dc481079c4a5db5b54d485991fa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.385073 28604 tablet_bootstrap.cc:492] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.386521 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 (DEFAULT_TABLE table=default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:00.393234 28150 leader_election.cc:341] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:00.393747 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ae10db451b4e3a8a85d1d987c447d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.394381 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 (DEFAULT_TABLE table=default.Table_4e83fe8e46644b8f8df871f6da4f0f1d [id=a1a30b8a5adb44b390e6bea2dd82c20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.388373 28473 tablet_service.cc:1467] Processing CreateTablet for tablet f54dc481079c4a5db5b54d485991fa65 (DEFAULT_TABLE table=default.Table_2088aba8de5d4a309d04f4f5d77d43bd [id=3e4bfd54a342491b86e978c08faa5c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.395486 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa994dfa54a47408bfb34bcfda9a415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.387843 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd98387c1cae45dcb29c2116b6141708" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:00.396534 28494 raft_consensus.cc:2454] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:00.397580 28150 leader_election.cc:302] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:00.398768 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54dc481079c4a5db5b54d485991fa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.401729 28637 raft_consensus.cc:2781] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.402024 28637 raft_consensus.cc:482] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.402191 28637 raft_consensus.cc:3037] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.403239 28333 tablet_service.cc:1467] Processing CreateTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 (DEFAULT_TABLE table=default.Table_0515e83b3d914915a67f1d30f123eb09 [id=f699bab9cb6a4364a6b371df69e16222]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.403821 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50237f7cc324e61a27ef6fe2dd7f071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.407809 28637 raft_consensus.cc:504] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.408592 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 567f15be6b9e4baaba9587c3634f72ad (DEFAULT_TABLE table=default.Table_666d072a7d2a4affa01a3cf64abf777c [id=c094b16f94d44132bea89dc256965971]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.409159 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567f15be6b9e4baaba9587c3634f72ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.409338 28637 leader_election.cc:288] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.410751 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd98387c1cae45dcb29c2116b6141708" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:00.411690 28150 leader_election.cc:341] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:00.412581 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 5d637087c60144aaaa9b0965110d280e (DEFAULT_TABLE table=default.Table_25378c7373494eb6a7e7e3bebd175e6b [id=500a3febdac042138ea31f6f3c8d02b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.412961 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:00.413151 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d637087c60144aaaa9b0965110d280e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.413746 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 90ee72bcd4654d06a91ccfb7c24869da from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.414629 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4e83fe8e46644b8f8df871f6da4f0f1d" } modify_external_catalogs: true
I20240503 14:12:00.414873 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4e83fe8e46644b8f8df871f6da4f0f1d" } modify_external_catalogs: true
I20240503 14:12:00.416812 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 38c0611cf7aa4ff3b4e7e6081558729b
I20240503 14:12:00.418938 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d37413e13809448ea8c94ae0e9efc618
I20240503 14:12:00.419606 28032 catalog_manager.cc:6554] ReplaceTablet: tablet d37413e13809448ea8c94ae0e9efc618 deleted and replaced by tablet d3d6e732eceb412b931c6387fb0c62bd
I20240503 14:12:00.420436 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c0611cf7aa4ff3b4e7e6081558729b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.422875 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa994dfa54a47408bfb34bcfda9a415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.429145 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd98387c1cae45dcb29c2116b6141708" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:00.429525 28494 raft_consensus.cc:3037] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.431651 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 (DEFAULT_TABLE table=default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.432294 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ae10db451b4e3a8a85d1d987c447d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.432807 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_36901c8428f346ae8117eb126334063b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.433655 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36901c8428f346ae8117eb126334063b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.435381 28604 tablet_bootstrap.cc:654] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.436671 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b2d7140d9a16416a97ef0b0b9b506468"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.437584 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2d7140d9a16416a97ef0b0b9b506468 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.437827 28622 raft_consensus.cc:482] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.438169 28622 raft_consensus.cc:504] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.439256 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4a16c898f66d4ad999025524253f572b" } modify_external_catalogs: true
I20240503 14:12:00.439551 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4a16c898f66d4ad999025524253f572b" } modify_external_catalogs: true
I20240503 14:12:00.441359 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9c4dd16181e4b0789779c3f25eaed1c" } modify_external_catalogs: true
I20240503 14:12:00.441748 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9c4dd16181e4b0789779c3f25eaed1c" } modify_external_catalogs: true
I20240503 14:12:00.442150 28622 leader_election.cc:288] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.447392 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_5cbe683faac947458db5ca9867db6c58 [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 2 got 0
I20240503 14:12:00.447749 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet fdccb370198b4eb69768aeed787fd682 (table default.Table_9c60ef813e134942b2bf7a67b5f6ecf4 [id=9f10df997dc04f05a4f052f20c4a2fc5]). Expected version 1 got 0
I20240503 14:12:00.449453 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet fd98387c1cae45dcb29c2116b6141708
I20240503 14:12:00.450224 28033 catalog_manager.cc:6554] ReplaceTablet: tablet fd98387c1cae45dcb29c2116b6141708 deleted and replaced by tablet a2bfdcb8e2b241ca9a6922fd8491e27b
I20240503 14:12:00.450830 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9c2ee09cf013422085a43000c85cfe57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.451628 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c2ee09cf013422085a43000c85cfe57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:00.453343 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2bfdcb8e2b241ca9a6922fd8491e27b has no consensus state
I20240503 14:12:00.453994 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7afc535ef1bb440493a9e65dc3c86fde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.454845 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7afc535ef1bb440493a9e65dc3c86fde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.457820 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2b1373375e074bbba74f7bb80cf43abf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.458511 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b1373375e074bbba74f7bb80cf43abf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.458912 28622 raft_consensus.cc:482] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.459199 28622 raft_consensus.cc:504] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.459936 28622 leader_election.cc:288] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.461764 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6270d949a6654e54b3cc252a54943112"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.462584 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6270d949a6654e54b3cc252a54943112 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.462467 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f64509bdd63e4324867d81972965e77b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.463204 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f64509bdd63e4324867d81972965e77b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.467382 28494 raft_consensus.cc:2454] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:00.473567 28190 tablet_service.cc:1467] Processing CreateTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 (DEFAULT_TABLE table=default.Table_0515e83b3d914915a67f1d30f123eb09 [id=f699bab9cb6a4364a6b371df69e16222]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.474345 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50237f7cc324e61a27ef6fe2dd7f071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.476660 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 5d637087c60144aaaa9b0965110d280e (DEFAULT_TABLE table=default.Table_25378c7373494eb6a7e7e3bebd175e6b [id=500a3febdac042138ea31f6f3c8d02b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.477193 28150 leader_election.cc:302] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:00.478627 28196 tablet_service.cc:1467] Processing CreateTablet for tablet f54dc481079c4a5db5b54d485991fa65 (DEFAULT_TABLE table=default.Table_2088aba8de5d4a309d04f4f5d77d43bd [id=3e4bfd54a342491b86e978c08faa5c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.479280 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54dc481079c4a5db5b54d485991fa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.481895 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 (DEFAULT_TABLE table=default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.482558 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ae10db451b4e3a8a85d1d987c447d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.483402 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 567f15be6b9e4baaba9587c3634f72ad (DEFAULT_TABLE table=default.Table_666d072a7d2a4affa01a3cf64abf777c [id=c094b16f94d44132bea89dc256965971]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.484005 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567f15be6b9e4baaba9587c3634f72ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.488121 28637 raft_consensus.cc:2781] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.488430 28637 raft_consensus.cc:686] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:00.488512 28634 raft_consensus.cc:482] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:00.489518 28542 tablet.cc:2369] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:00.488802 28634 raft_consensus.cc:504] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.493988 28637 consensus_queue.cc:237] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.501621 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d637087c60144aaaa9b0965110d280e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.509604 28634 leader_election.cc:288] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:00.525745 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 5d637087c60144aaaa9b0965110d280e (DEFAULT_TABLE table=default.Table_25378c7373494eb6a7e7e3bebd175e6b [id=500a3febdac042138ea31f6f3c8d02b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.526450 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d637087c60144aaaa9b0965110d280e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.532238 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 567f15be6b9e4baaba9587c3634f72ad (DEFAULT_TABLE table=default.Table_666d072a7d2a4affa01a3cf64abf777c [id=c094b16f94d44132bea89dc256965971]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.532979 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567f15be6b9e4baaba9587c3634f72ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.535727 28610 raft_consensus.cc:482] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.536007 28610 raft_consensus.cc:504] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.536689 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 67ec68f75fa2435698383a03874a5bae (DEFAULT_TABLE table=default.Table_a9c4dd16181e4b0789779c3f25eaed1c [id=18925bb0895246988de55da2f108454b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.537396 28610 leader_election.cc:288] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.538429 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc56c4591c437da5b9686b726686b2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:00.538564 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc56c4591c437da5b9686b726686b2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:00.538741 28357 raft_consensus.cc:2454] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:00.538920 28494 raft_consensus.cc:2454] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:00.539501 28150 leader_election.cc:302] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:00.540352 28610 raft_consensus.cc:2781] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.540549 28610 raft_consensus.cc:482] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.540678 28610 raft_consensus.cc:3037] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.552174 28610 raft_consensus.cc:504] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.553972 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc56c4591c437da5b9686b726686b2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:00.554327 28357 raft_consensus.cc:3037] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.558660 28357 raft_consensus.cc:2454] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:00.559453 28150 leader_election.cc:302] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:00.560456 28471 tablet_service.cc:1467] Processing CreateTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 (DEFAULT_TABLE table=default.Table_0515e83b3d914915a67f1d30f123eb09 [id=f699bab9cb6a4364a6b371df69e16222]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.561070 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50237f7cc324e61a27ef6fe2dd7f071. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:00.570489 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 160949us
I20240503 14:12:00.573141 27851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu with pid 27859
I20240503 14:12:00.590390 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ec68f75fa2435698383a03874a5bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.592777 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.593206 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.596761 28653 tablet_replica.cc:330] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:00.601397 28653 raft_consensus.cc:2229] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.601972 28653 raft_consensus.cc:2258] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.603322 28610 leader_election.cc:288] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.603574 28610 raft_consensus.cc:482] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.603763 28610 raft_consensus.cc:504] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.604694 28610 leader_election.cc:288] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.604908 28610 raft_consensus.cc:2781] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.605064 28610 raft_consensus.cc:686] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:00.605419 28610 consensus_queue.cc:237] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.607715 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc56c4591c437da5b9686b726686b2" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:00.608206 28493 raft_consensus.cc:2454] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:00.609010 28290 leader_election.cc:302] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:00.609584 28622 raft_consensus.cc:2741] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 14:12:00.609820 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccb370198b4eb69768aeed787fd682" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:00.610720 28290 leader_election.cc:341] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:00.611872 28653 ts_tablet_manager.cc:1875] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:00.613520 28403 heartbeater.cc:644] Failed to heartbeat to 127.27.50.254:36571 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.27.50.254:36571 (error 108)
I20240503 14:12:00.615710 28614 tablet_bootstrap.cc:492] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.615931 28614 ts_tablet_manager.cc:1367] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.237s user 0.102s sys 0.087s
I20240503 14:12:00.616981 28614 raft_consensus.cc:348] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.617242 28614 raft_consensus.cc:374] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.617357 28614 raft_consensus.cc:729] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.617662 28614 consensus_queue.cc:260] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.619230 28614 ts_tablet_manager.cc:1398] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:00.619639 28614 tablet_bootstrap.cc:492] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.625607 28614 tablet_bootstrap.cc:654] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:00.631698 28017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.50.254:36571 (error 108)
W20240503 14:12:00.634100 28645 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:12:00.614327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.50.254:36571 (local address 127.27.50.194:34413)
0503 14:12:00.615250 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0503 14:12:00.615274 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 14:12:00.615418 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:12:00.633001 (+ 17583us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":790,"threads_started":1}
W20240503 14:12:00.639267 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 71598d0a44384a3b93dec30d6c60ecd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71598d0a44384a3b93dec30d6c60ecd5 already in progress: creating tablet
I20240503 14:12:00.641100 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.642345 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet d37413e13809448ea8c94ae0e9efc618 with delete_type TABLET_DATA_DELETED (replaced by tablet d3d6e732eceb412b931c6387fb0c62bd) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.642745 28603 tablet_replica.cc:330] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:00.643348 28603 raft_consensus.cc:2229] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.643839 28603 raft_consensus.cc:2258] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.645099 28028 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d3d6e732eceb412b931c6387fb0c62bd
I20240503 14:12:00.645710 28603 ts_tablet_manager.cc:1875] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.648290 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_5cbe683faac947458db5ca9867db6c58 [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 2 got 0
I20240503 14:12:00.648165 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet d37413e13809448ea8c94ae0e9efc618 with delete_type TABLET_DATA_DELETED (replaced by tablet d3d6e732eceb412b931c6387fb0c62bd) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.651088 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.651818 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3dd432e943764d9296fada2cf3c0ca47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.652643 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dd432e943764d9296fada2cf3c0ca47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.654366 28603 ts_tablet_manager.cc:1888] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.654556 28603 log.cc:1196] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/d37413e13809448ea8c94ae0e9efc618
I20240503 14:12:00.654503 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5f10f98658754315a6cc9b1d7ffc19cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.655097 28603 ts_tablet_manager.cc:1909] T d37413e13809448ea8c94ae0e9efc618 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:00.655234 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f10f98658754315a6cc9b1d7ffc19cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.656615 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.657169 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet fd98387c1cae45dcb29c2116b6141708 with delete_type TABLET_DATA_DELETED (replaced by tablet a2bfdcb8e2b241ca9a6922fd8491e27b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.657189 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_50ca74d27bab49ff9f56b8e584929e8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.657558 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d0a58592c946bfa57a5698add39985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:00.657879 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50ca74d27bab49ff9f56b8e584929e8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.657935 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet e20b99c022ea437181f8d3221b952932 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.658344 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.658967 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet fd98387c1cae45dcb29c2116b6141708 with delete_type TABLET_DATA_DELETED (replaced by tablet a2bfdcb8e2b241ca9a6922fd8491e27b) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.659607 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_35f27fac8e5342d28c1883f89c3491c1" } new_table_name: "default.Table_0f8086df11104655afdf191191957937" modify_external_catalogs: true
I20240503 14:12:00.660619 28604 tablet_bootstrap.cc:492] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:00.662191 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2088aba8de5d4a309d04f4f5d77d43bd" } new_table_name: "default.Table_9289210d002542b78ccb48780cf16018" modify_external_catalogs: true
I20240503 14:12:00.662636 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d0a58592c946bfa57a5698add39985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.663534 28604 ts_tablet_manager.cc:1367] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.278s user 0.011s sys 0.011s
I20240503 14:12:00.664652 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 67ec68f75fa2435698383a03874a5bae
I20240503 14:12:00.665617 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5cbe683faac947458db5ca9867db6c58" } new_table_name: "default.Table_fbe6764ce0f84db08b3d9ef65525d733" modify_external_catalogs: true
I20240503 14:12:00.666503 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet e20b99c022ea437181f8d3221b952932 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.668002 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6aa994dfa54a47408bfb34bcfda9a415
I20240503 14:12:00.668813 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ad66d05a032c41f0b4e2ec45e9044b31" } new_table_name: "default.Table_30f21f3fef874fdf94af3d3d043de7d7" modify_external_catalogs: true
I20240503 14:12:00.669293 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccb370198b4eb69768aeed787fd682" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:00.669775 28217 raft_consensus.cc:2454] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:00.669903 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.670428 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc56c4591c437da5b9686b726686b2" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:00.671170 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_666d072a7d2a4affa01a3cf64abf777c" } modify_external_catalogs: true
I20240503 14:12:00.671320 28290 leader_election.cc:302] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:00.671384 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_666d072a7d2a4affa01a3cf64abf777c" } modify_external_catalogs: true
I20240503 14:12:00.671679 28622 raft_consensus.cc:2781] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.671813 28622 raft_consensus.cc:482] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.671914 28622 raft_consensus.cc:3037] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:00.672935 28540 heartbeater.cc:644] Failed to heartbeat to 127.27.50.254:36571 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.27.50.254:36571 (error 108)
I20240503 14:12:00.675091 28622 raft_consensus.cc:504] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.675978 28622 leader_election.cc:288] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.682013 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 42d0a58592c946bfa57a5698add39985 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.683331 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0515e83b3d914915a67f1d30f123eb09" } modify_external_catalogs: true
I20240503 14:12:00.683596 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0515e83b3d914915a67f1d30f123eb09" } modify_external_catalogs: true
I20240503 14:12:00.683396 28604 raft_consensus.cc:348] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.687038 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccb370198b4eb69768aeed787fd682" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:00.687312 28217 raft_consensus.cc:3037] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.686604 28604 raft_consensus.cc:374] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.688222 28604 raft_consensus.cc:729] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.689132 28661 tablet_replica.cc:330] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:00.689445 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccb370198b4eb69768aeed787fd682" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:00.690701 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ec68f75fa2435698383a03874a5bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.691108 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ec68f75fa2435698383a03874a5bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.691695 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:00.692780 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 133350us
I20240503 14:12:00.693290 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20b99c022ea437181f8d3221b952932" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:00.693603 28357 raft_consensus.cc:2454] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:00.693656 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.693728 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfc56c4591c437da5b9686b726686b2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:00.693998 28493 raft_consensus.cc:3037] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.694909 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet d37413e13809448ea8c94ae0e9efc618 with delete_type TABLET_DATA_DELETED (replaced by tablet d3d6e732eceb412b931c6387fb0c62bd) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.695796 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.693059 28604 consensus_queue.cc:260] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.696391 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet fd98387c1cae45dcb29c2116b6141708 with delete_type TABLET_DATA_DELETED (replaced by tablet a2bfdcb8e2b241ca9a6922fd8491e27b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.696794 28494 raft_consensus.cc:3037] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.696981 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ec68f75fa2435698383a03874a5bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.697445 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.697763 28493 raft_consensus.cc:2454] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:00.699824 28494 raft_consensus.cc:2454] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:00.701948 28290 leader_election.cc:302] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:00.702742 28603 tablet_replica.cc:330] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:00.703068 28217 raft_consensus.cc:2454] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:00.703116 28603 raft_consensus.cc:2229] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.705480 28622 raft_consensus.cc:2781] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.709466 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccb370198b4eb69768aeed787fd682" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:00.700300 28661 raft_consensus.cc:2229] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:00.710057 28263 heartbeater.cc:644] Failed to heartbeat to 127.27.50.254:36571 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.27.50.254:36571 (error 108)
I20240503 14:12:00.710220 28661 raft_consensus.cc:2258] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:00.710659 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 38c0611cf7aa4ff3b4e7e6081558729b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38c0611cf7aa4ff3b4e7e6081558729b already in progress: creating tablet
I20240503 14:12:00.711565 28604 ts_tablet_manager.cc:1398] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.047s user 0.004s sys 0.000s
I20240503 14:12:00.711957 28604 tablet_bootstrap.cc:492] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.713383 28653 ts_tablet_manager.cc:1888] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.713605 28653 log.cc:1196] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:00.713944 28604 tablet_bootstrap.cc:654] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.714164 28653 ts_tablet_manager.cc:1909] T 71598d0a44384a3b93dec30d6c60ecd5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:00.714221 28603 raft_consensus.cc:2258] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.707540 28427 leader_election.cc:302] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:00.709482 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccb370198b4eb69768aeed787fd682" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:00.715478 28634 raft_consensus.cc:2781] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.715639 28634 raft_consensus.cc:482] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.715734 28634 raft_consensus.cc:3037] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.716024 28603 ts_tablet_manager.cc:1875] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.718722 28634 raft_consensus.cc:504] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.719632 28634 leader_election.cc:288] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:00.715476 28494 raft_consensus.cc:2379] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
W20240503 14:12:00.720808 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 71598d0a44384a3b93dec30d6c60ecd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71598d0a44384a3b93dec30d6c60ecd5 already in progress: creating tablet
W20240503 14:12:00.721161 28017 consensus_peers.cc:485] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 -> Peer bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571): Couldn't send request to peer bb6a7aee99214df4b02858ddae025b14. Status: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:00.721333 28600 tablet_bootstrap.cc:492] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.721544 28600 ts_tablet_manager.cc:1367] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.389s user 0.009s sys 0.006s
W20240503 14:12:00.721540 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 38c0611cf7aa4ff3b4e7e6081558729b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38c0611cf7aa4ff3b4e7e6081558729b already in progress: creating tablet
W20240503 14:12:00.721781 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet fd98387c1cae45dcb29c2116b6141708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd98387c1cae45dcb29c2116b6141708 already in progress: creating tablet
I20240503 14:12:00.722553 28600 raft_consensus.cc:348] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.722818 28600 raft_consensus.cc:374] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.722914 28600 raft_consensus.cc:729] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.723251 28600 consensus_queue.cc:260] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.723898 28600 ts_tablet_manager.cc:1398] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:12:00.724262 28600 tablet_bootstrap.cc:492] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.725708 28661 ts_tablet_manager.cc:1875] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.726186 28600 tablet_bootstrap.cc:654] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.732264 28653 tablet_replica.cc:330] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:00.732556 28653 raft_consensus.cc:2229] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.732766 28653 raft_consensus.cc:2258] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.733431 28653 ts_tablet_manager.cc:1875] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.739048 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20b99c022ea437181f8d3221b952932" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:00.739351 28217 raft_consensus.cc:2454] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:00.740022 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 3826594d300342c7a59e0eee989a873f (table default.Table_ad66d05a032c41f0b4e2ec45e9044b31 [id=208684b1dfd044c6af3b7b4fa83b8c2b]). Expected version 2 got 0
I20240503 14:12:00.740041 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20b99c022ea437181f8d3221b952932" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:00.740288 28217 raft_consensus.cc:3037] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.742383 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet f35e87db1c2a42a79b426b2e7767367a from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:00.742861 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c6782582fb543f4b621569343995dfc consensus state has no leader
I20240503 14:12:00.743014 28217 raft_consensus.cc:2454] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:00.743201 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet 6aa994dfa54a47408bfb34bcfda9a415 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:00.743657 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f54dc481079c4a5db5b54d485991fa65 consensus state has no leader
I20240503 14:12:00.743953 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 71598d0a44384a3b93dec30d6c60ecd5 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:00.744403 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bfc56c4591c437da5b9686b726686b2 consensus state has no leader
W20240503 14:12:00.745914 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3826594d300342c7a59e0eee989a873f consensus state has no leader
I20240503 14:12:00.745811 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20b99c022ea437181f8d3221b952932" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:00.746166 28356 raft_consensus.cc:3037] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.746186 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 87ae10db451b4e3a8a85d1d987c447d9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.749547 28356 raft_consensus.cc:2454] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:00.751837 28427 leader_election.cc:302] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:00.751994 28622 raft_consensus.cc:686] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:00.752776 28634 raft_consensus.cc:2781] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.752640 28622 consensus_queue.cc:237] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.754335 28614 tablet_bootstrap.cc:492] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.754500 28600 tablet_bootstrap.cc:492] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.754527 28614 ts_tablet_manager.cc:1367] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.135s user 0.040s sys 0.064s
I20240503 14:12:00.754765 28600 ts_tablet_manager.cc:1367] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240503 14:12:00.755426 28614 raft_consensus.cc:348] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.755663 28614 raft_consensus.cc:374] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.755791 28614 raft_consensus.cc:729] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.755602 28600 raft_consensus.cc:348] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.756071 28600 raft_consensus.cc:374] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.756208 28600 raft_consensus.cc:729] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.756039 28614 consensus_queue.cc:260] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.756600 28600 consensus_queue.cc:260] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.758016 28614 ts_tablet_manager.cc:1398] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:00.759348 28614 tablet_bootstrap.cc:492] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.761229 28603 ts_tablet_manager.cc:1888] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.761618 28600 ts_tablet_manager.cc:1398] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:00.763859 28600 tablet_bootstrap.cc:492] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.765451 28150 leader_election.cc:302] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:00.765759 28651 raft_consensus.cc:2726] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:00.768306 28473 tablet_service.cc:1467] Processing CreateTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 (DEFAULT_TABLE table=default.Table_36901c8428f346ae8117eb126334063b [id=15661374a8c648338c1386ad0b207148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.768865 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9713865e09349ce9f9ae1d4297da2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.769348 28604 tablet_bootstrap.cc:492] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:00.769544 28604 ts_tablet_manager.cc:1367] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20240503 14:12:00.770807 28600 tablet_bootstrap.cc:654] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.770949 28471 tablet_service.cc:1467] Processing CreateTablet for tablet b166fe6366ba496a869af1b106fe9d8e (DEFAULT_TABLE table=default.Table_b2d7140d9a16416a97ef0b0b9b506468 [id=548632b6d08e4435ad561b074e5e5b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.771436 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b166fe6366ba496a869af1b106fe9d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.773089 28194 tablet_service.cc:1467] Processing CreateTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 (DEFAULT_TABLE table=default.Table_36901c8428f346ae8117eb126334063b [id=15661374a8c648338c1386ad0b207148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.773543 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9713865e09349ce9f9ae1d4297da2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.773757 28189 tablet_service.cc:1467] Processing CreateTablet for tablet b166fe6366ba496a869af1b106fe9d8e (DEFAULT_TABLE table=default.Table_b2d7140d9a16416a97ef0b0b9b506468 [id=548632b6d08e4435ad561b074e5e5b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.774372 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b166fe6366ba496a869af1b106fe9d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.776029 28196 tablet_service.cc:1467] Processing CreateTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 (DEFAULT_TABLE table=default.Table_7afc535ef1bb440493a9e65dc3c86fde [id=52c4ea4489f24aa19d1671b4a5f88808]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.776657 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 (DEFAULT_TABLE table=default.Table_6270d949a6654e54b3cc252a54943112 [id=ada8758d8af14ac7bc31969934efb7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.777087 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba629915ef44e0882beee2612e285c1 (DEFAULT_TABLE table=default.Table_9c2ee09cf013422085a43000c85cfe57 [id=bfa896d86f6a447089c787a2c220ab9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.777436 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe5d280a1654d80a2176c1def459e2e (DEFAULT_TABLE table=default.Table_2b1373375e074bbba74f7bb80cf43abf [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.777838 28197 tablet_service.cc:1467] Processing CreateTablet for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (DEFAULT_TABLE table=default.Table_059ac6056dc04f58b62976bc4cb523a2 [id=bd615ce41c41475fa37c3d6de18ece3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.778088 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85b6db1a21c4bca80c8bdec17e45c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.778493 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe5d280a1654d80a2176c1def459e2e (DEFAULT_TABLE table=default.Table_2b1373375e074bbba74f7bb80cf43abf [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.778916 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe5d280a1654d80a2176c1def459e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.780494 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 71598d0a44384a3b93dec30d6c60ecd5 (table default.Table_2be03633b8854e03955f07503a5309a5 [id=7907cfab39b74beba4ceddd1762bffcd]) successfully deleted
I20240503 14:12:00.781389 28603 log.cc:1196] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/42d0a58592c946bfa57a5698add39985
I20240503 14:12:00.781666 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba629915ef44e0882beee2612e285c1 (DEFAULT_TABLE table=default.Table_9c2ee09cf013422085a43000c85cfe57 [id=bfa896d86f6a447089c787a2c220ab9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.781965 28603 ts_tablet_manager.cc:1909] T 42d0a58592c946bfa57a5698add39985 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:00.782153 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba629915ef44e0882beee2612e285c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.761318 28614 tablet_bootstrap.cc:654] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.764761 28329 tablet_service.cc:1467] Processing CreateTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 (DEFAULT_TABLE table=default.Table_36901c8428f346ae8117eb126334063b [id=15661374a8c648338c1386ad0b207148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.782549 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9713865e09349ce9f9ae1d4297da2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.783321 28603 tablet_replica.cc:330] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:00.783541 28603 raft_consensus.cc:2229] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.783689 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bfdcb8e2b241ca9a6922fd8491e27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.783742 28603 raft_consensus.cc:2258] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.784395 28603 ts_tablet_manager.cc:1875] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.786095 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba629915ef44e0882beee2612e285c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.787204 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe5d280a1654d80a2176c1def459e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.775394 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe5d280a1654d80a2176c1def459e2e (DEFAULT_TABLE table=default.Table_2b1373375e074bbba74f7bb80cf43abf [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.775784 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba629915ef44e0882beee2612e285c1 (DEFAULT_TABLE table=default.Table_9c2ee09cf013422085a43000c85cfe57 [id=bfa896d86f6a447089c787a2c220ab9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.776199 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 (DEFAULT_TABLE table=default.Table_6270d949a6654e54b3cc252a54943112 [id=ada8758d8af14ac7bc31969934efb7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.776865 28335 tablet_service.cc:1467] Processing CreateTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 (DEFAULT_TABLE table=default.Table_7afc535ef1bb440493a9e65dc3c86fde [id=52c4ea4489f24aa19d1671b4a5f88808]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.778692 28334 tablet_service.cc:1467] Processing CreateTablet for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (DEFAULT_TABLE table=default.Table_059ac6056dc04f58b62976bc4cb523a2 [id=bd615ce41c41475fa37c3d6de18ece3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.787930 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe5d280a1654d80a2176c1def459e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.788306 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6106d52df3c54c2bb4abadd8605cd912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.788391 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bfdcb8e2b241ca9a6922fd8491e27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.790812 28473 tablet_service.cc:1467] Processing CreateTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 (DEFAULT_TABLE table=default.Table_7afc535ef1bb440493a9e65dc3c86fde [id=52c4ea4489f24aa19d1671b4a5f88808]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.791373 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85b6db1a21c4bca80c8bdec17e45c59. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:00.791819 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_fbe6764ce0f84db08b3d9ef65525d733 [id=3a24797cc53b439da44cc77025fa9f92], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8bfc56c4591c437da5b9686b726686b2 consensus state has no leader
I20240503 14:12:00.791875 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85b6db1a21c4bca80c8bdec17e45c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.796537 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6106d52df3c54c2bb4abadd8605cd912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.796470 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 (DEFAULT_TABLE table=default.Table_6270d949a6654e54b3cc252a54943112 [id=ada8758d8af14ac7bc31969934efb7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.796929 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6106d52df3c54c2bb4abadd8605cd912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.800232 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 42d0a58592c946bfa57a5698add39985 (table default.Table_0a89e43de47c4ee2ba100e7dbf451d84 [id=72babfde5906464b9a4e55e85c5db68e]) successfully deleted
I20240503 14:12:00.800374 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba629915ef44e0882beee2612e285c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.801518 28468 tablet_service.cc:1467] Processing CreateTablet for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (DEFAULT_TABLE table=default.Table_059ac6056dc04f58b62976bc4cb523a2 [id=bd615ce41c41475fa37c3d6de18ece3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.802000 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bfdcb8e2b241ca9a6922fd8491e27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.779366 28330 tablet_service.cc:1467] Processing CreateTablet for tablet b166fe6366ba496a869af1b106fe9d8e (DEFAULT_TABLE table=default.Table_b2d7140d9a16416a97ef0b0b9b506468 [id=548632b6d08e4435ad561b074e5e5b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:00.779569 28662 raft_consensus.cc:482] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.805002 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b166fe6366ba496a869af1b106fe9d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:00.804761 28662 raft_consensus.cc:504] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.805959 28662 leader_election.cc:288] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.806974 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ee72bcd4654d06a91ccfb7c24869da" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:00.807236 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ee72bcd4654d06a91ccfb7c24869da" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:00.807966 28290 leader_election.cc:341] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:00.810621 28604 raft_consensus.cc:348] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.810876 28604 raft_consensus.cc:374] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.810962 28604 raft_consensus.cc:729] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.811239 28604 consensus_queue.cc:260] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.818521 28634 raft_consensus.cc:686] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:00.819201 28634 consensus_queue.cc:237] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:00.819362 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet fdccb370198b4eb69768aeed787fd682 (table default.Table_9c60ef813e134942b2bf7a67b5f6ecf4 [id=9f10df997dc04f05a4f052f20c4a2fc5]). Expected version 1 got 0
I20240503 14:12:00.819717 28033 catalog_manager.cc:5484] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:00.820700 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_fbe6764ce0f84db08b3d9ef65525d733 [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 3 got 0
I20240503 14:12:00.821062 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 3826594d300342c7a59e0eee989a873f (table default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b]). Expected version 3 got 0
I20240503 14:12:00.822054 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9c2ee09cf013422085a43000c85cfe57" } new_table_name: "default.Table_5cf8c9529d1f4adb9fbee8eb11985651" modify_external_catalogs: true
I20240503 14:12:00.824864 28653 ts_tablet_manager.cc:1888] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.828238 28604 ts_tablet_manager.cc:1398] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.059s user 0.002s sys 0.001s
I20240503 14:12:00.828697 28604 tablet_bootstrap.cc:492] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.834403 28604 tablet_bootstrap.cc:654] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.834475 28217 raft_consensus.cc:2454] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:00.835067 28290 leader_election.cc:302] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:00.835630 28662 raft_consensus.cc:2781] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.835763 28662 raft_consensus.cc:482] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.835855 28662 raft_consensus.cc:3037] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.838646 28662 raft_consensus.cc:504] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.839896 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ee72bcd4654d06a91ccfb7c24869da" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:00.840939 28662 leader_election.cc:288] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.842013 28614 tablet_bootstrap.cc:492] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.842278 28614 ts_tablet_manager.cc:1367] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.008s
I20240503 14:12:00.843127 28614 raft_consensus.cc:348] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.843365 28614 raft_consensus.cc:374] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.843446 28614 raft_consensus.cc:729] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.843797 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ee72bcd4654d06a91ccfb7c24869da" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:00.844437 28290 leader_election.cc:341] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:00.844527 28661 ts_tablet_manager.cc:1888] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.844684 28661 log.cc:1196] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/fd98387c1cae45dcb29c2116b6141708
I20240503 14:12:00.844730 28614 consensus_queue.cc:260] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.845191 28661 ts_tablet_manager.cc:1909] T fd98387c1cae45dcb29c2116b6141708 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:00.849272 28614 ts_tablet_manager.cc:1398] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:00.851794 28614 tablet_bootstrap.cc:492] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.853026 28217 raft_consensus.cc:3037] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.853327 28661 tablet_replica.cc:330] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:00.853598 28661 raft_consensus.cc:2229] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.853807 28661 raft_consensus.cc:2258] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.854595 28614 tablet_bootstrap.cc:654] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.854802 28661 ts_tablet_manager.cc:1875] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.855945 28217 raft_consensus.cc:2454] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:00.856737 28653 log.cc:1196] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/d37413e13809448ea8c94ae0e9efc618
I20240503 14:12:00.856524 28290 leader_election.cc:302] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:00.857393 28653 ts_tablet_manager.cc:1909] T d37413e13809448ea8c94ae0e9efc618 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:00.857419 28662 raft_consensus.cc:2781] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.857600 28662 raft_consensus.cc:686] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:00.857934 28662 consensus_queue.cc:237] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.859637 28651 raft_consensus.cc:482] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.859822 28651 raft_consensus.cc:504] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.860761 28651 leader_election.cc:288] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.861340 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3826594d300342c7a59e0eee989a873f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:00.861392 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3826594d300342c7a59e0eee989a873f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:00.861611 28357 raft_consensus.cc:2454] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:00.862094 28603 ts_tablet_manager.cc:1888] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.862236 28603 log.cc:1196] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e20b99c022ea437181f8d3221b952932
I20240503 14:12:00.862294 28150 leader_election.cc:302] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:00.862454 28604 tablet_bootstrap.cc:492] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:00.862610 28603 ts_tablet_manager.cc:1909] T e20b99c022ea437181f8d3221b952932 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:00.862643 28150 leader_election.cc:341] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:00.862663 28604 ts_tablet_manager.cc:1367] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240503 14:12:00.863173 28651 raft_consensus.cc:2781] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.863276 28651 raft_consensus.cc:482] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.863369 28651 raft_consensus.cc:3037] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.863639 28603 tablet_replica.cc:330] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:00.863596 28604 raft_consensus.cc:348] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.863831 28604 raft_consensus.cc:374] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.863922 28604 raft_consensus.cc:729] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.864105 28653 tablet_replica.cc:330] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:00.864264 28604 consensus_queue.cc:260] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.865207 28653 raft_consensus.cc:2229] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.865433 28653 raft_consensus.cc:2258] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.866303 28653 ts_tablet_manager.cc:1875] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.866817 28651 raft_consensus.cc:504] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.867592 28651 leader_election.cc:288] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:00.868561 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3826594d300342c7a59e0eee989a873f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:00.868800 28357 raft_consensus.cc:3037] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.869200 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3826594d300342c7a59e0eee989a873f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:00.870357 28604 ts_tablet_manager.cc:1398] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:00.871624 28357 raft_consensus.cc:2454] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:00.873615 28653 ts_tablet_manager.cc:1888] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.873780 28653 log.cc:1196] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/38c0611cf7aa4ff3b4e7e6081558729b
I20240503 14:12:00.874082 28150 leader_election.cc:302] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
W20240503 14:12:00.874590 28150 leader_election.cc:341] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:00.875046 28651 raft_consensus.cc:2781] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.875116 28604 tablet_bootstrap.cc:492] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.876982 28651 raft_consensus.cc:686] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:00.877602 28651 consensus_queue.cc:237] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.878887 28604 tablet_bootstrap.cc:654] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.874205 28653 ts_tablet_manager.cc:1909] T 38c0611cf7aa4ff3b4e7e6081558729b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:00.879813 28603 raft_consensus.cc:2229] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:00.882974 28603 pending_rounds.cc:62] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:00.882102 28600 tablet_bootstrap.cc:492] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.883464 28603 pending_rounds.cc:68] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597653437665280 op_type: NO_OP noop_request { }
I20240503 14:12:00.883702 28603 raft_consensus.cc:2866] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:00.883881 28603 raft_consensus.cc:2258] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.887802 28661 ts_tablet_manager.cc:1888] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.888016 28661 log.cc:1196] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/42d0a58592c946bfa57a5698add39985
I20240503 14:12:00.888480 28661 ts_tablet_manager.cc:1909] T 42d0a58592c946bfa57a5698add39985 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:00.888525 28600 ts_tablet_manager.cc:1367] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.001s
I20240503 14:12:00.889462 28600 raft_consensus.cc:348] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.889935 28600 raft_consensus.cc:374] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.890075 28600 raft_consensus.cc:729] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.890421 28600 consensus_queue.cc:260] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.890599 28603 ts_tablet_manager.cc:1875] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.892791 28600 ts_tablet_manager.cc:1398] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:00.894012 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet fd98387c1cae45dcb29c2116b6141708 (table default.Table_059ac6056dc04f58b62976bc4cb523a2 [id=bd615ce41c41475fa37c3d6de18ece3a]) successfully deleted
I20240503 14:12:00.895042 28600 tablet_bootstrap.cc:492] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.896292 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 567f15be6b9e4baaba9587c3634f72ad
I20240503 14:12:00.893371 28653 tablet_replica.cc:330] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:00.898679 28603 ts_tablet_manager.cc:1888] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:00.899003 28634 raft_consensus.cc:482] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.899230 28634 raft_consensus.cc:504] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.899955 28634 leader_election.cc:288] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:00.900456 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a50237f7cc324e61a27ef6fe2dd7f071
I20240503 14:12:00.900861 28603 log.cc:1196] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/90ee72bcd4654d06a91ccfb7c24869da
I20240503 14:12:00.901103 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ee72bcd4654d06a91ccfb7c24869da" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:00.901379 28603 ts_tablet_manager.cc:1909] T 90ee72bcd4654d06a91ccfb7c24869da P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:00.901409 28217 raft_consensus.cc:2379] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
W20240503 14:12:00.902892 28427 leader_election.cc:341] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 90ee72bcd4654d06a91ccfb7c24869da
I20240503 14:12:00.903124 28427 leader_election.cc:302] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:00.903376 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e20b99c022ea437181f8d3221b952932 (table default.Table_c75bd5158ad74ccd9cc5ffd60749907a [id=d2b505b742e64fa6948bf41c834d8902]) successfully deleted
I20240503 14:12:00.904146 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 38c0611cf7aa4ff3b4e7e6081558729b (table default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e [id=e6b422e744c14453a7e7c4b28c1eaac1]) successfully deleted
I20240503 14:12:00.904167 28634 raft_consensus.cc:3037] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.904302 28661 tablet_replica.cc:330] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:00.904428 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 42d0a58592c946bfa57a5698add39985 (table default.Table_0a89e43de47c4ee2ba100e7dbf451d84 [id=72babfde5906464b9a4e55e85c5db68e]) successfully deleted
I20240503 14:12:00.904533 28661 raft_consensus.cc:2229] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:00.904857 28653 raft_consensus.cc:2229] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:00.905135 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 90ee72bcd4654d06a91ccfb7c24869da (table default.Table_bc04ce776cfb4771977f33fa236ea77f [id=a9b7085b3a83482a91e6a8c888956a4e]) successfully deleted
I20240503 14:12:00.905391 28653 pending_rounds.cc:62] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:00.905619 28653 pending_rounds.cc:68] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597651952312320 op_type: NO_OP noop_request { }
I20240503 14:12:00.905740 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f15be6b9e4baaba9587c3634f72ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.905994 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.906867 28653 raft_consensus.cc:2866] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:00.907059 28653 raft_consensus.cc:2258] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.907142 28634 raft_consensus.cc:2726] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:00.908206 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f15be6b9e4baaba9587c3634f72ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.908452 28653 ts_tablet_manager.cc:1875] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.908497 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.909480 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f15be6b9e4baaba9587c3634f72ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.909482 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:00.909889 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 67ec68f75fa2435698383a03874a5bae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67ec68f75fa2435698383a03874a5bae already in progress: creating tablet
I20240503 14:12:00.910192 28661 pending_rounds.cc:62] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:00.910368 28661 pending_rounds.cc:68] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597653281443840 op_type: NO_OP noop_request { }
I20240503 14:12:00.910611 28661 raft_consensus.cc:2866] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:00.910748 28661 raft_consensus.cc:2258] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:00.911742 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6aa994dfa54a47408bfb34bcfda9a415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6aa994dfa54a47408bfb34bcfda9a415 already in progress: creating tablet
I20240503 14:12:00.912379 28600 tablet_bootstrap.cc:654] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:00.912961 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 567f15be6b9e4baaba9587c3634f72ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567f15be6b9e4baaba9587c3634f72ad already in progress: creating tablet
W20240503 14:12:00.914134 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a50237f7cc324e61a27ef6fe2dd7f071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a50237f7cc324e61a27ef6fe2dd7f071 already in progress: creating tablet
I20240503 14:12:00.915297 28661 ts_tablet_manager.cc:1875] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.915936 28653 ts_tablet_manager.cc:1888] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:00.916172 28653 log.cc:1196] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/fd98387c1cae45dcb29c2116b6141708
I20240503 14:12:00.916708 28653 ts_tablet_manager.cc:1909] T fd98387c1cae45dcb29c2116b6141708 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:00.919924 28614 tablet_bootstrap.cc:492] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.920194 28614 ts_tablet_manager.cc:1367] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20240503 14:12:00.921180 28614 raft_consensus.cc:348] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.921453 28614 raft_consensus.cc:374] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.921576 28614 raft_consensus.cc:729] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.921697 28600 tablet_bootstrap.cc:492] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.921929 28604 tablet_bootstrap.cc:492] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:00.921939 28600 ts_tablet_manager.cc:1367] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.003s
I20240503 14:12:00.921864 28614 consensus_queue.cc:260] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.921948 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9d94370dfab6449ea9ad79c5530f078a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.922163 28604 ts_tablet_manager.cc:1367] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
I20240503 14:12:00.923075 28604 raft_consensus.cc:348] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.923012 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_31bf8350c35e43409cfb20708c238df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.923321 28604 raft_consensus.cc:374] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.923418 28604 raft_consensus.cc:729] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.923219 28600 raft_consensus.cc:348] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.923672 28614 ts_tablet_manager.cc:1398] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:00.923748 28604 consensus_queue.cc:260] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:00.923918 28600 raft_consensus.cc:374] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.924116 28614 tablet_bootstrap.cc:492] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.924160 28600 raft_consensus.cc:729] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.924263 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet fd98387c1cae45dcb29c2116b6141708 (table default.Table_059ac6056dc04f58b62976bc4cb523a2 [id=bd615ce41c41475fa37c3d6de18ece3a]) successfully deleted
I20240503 14:12:00.924422 28604 ts_tablet_manager.cc:1398] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240503 14:12:00.924614 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d94370dfab6449ea9ad79c5530f078a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.924885 28600 consensus_queue.cc:260] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:00.926008 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31bf8350c35e43409cfb20708c238df5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.926079 28614 tablet_bootstrap.cc:654] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.927255 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_976081c8371b4dc68cb226625ef1b6c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.927907 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_976081c8371b4dc68cb226625ef1b6c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.931133 28604 tablet_bootstrap.cc:492] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:00.936074 28600 ts_tablet_manager.cc:1398] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240503 14:12:00.936864 28604 tablet_bootstrap.cc:654] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:00.937177 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 67ec68f75fa2435698383a03874a5bae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67ec68f75fa2435698383a03874a5bae already in progress: creating tablet
W20240503 14:12:00.938813 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6aa994dfa54a47408bfb34bcfda9a415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6aa994dfa54a47408bfb34bcfda9a415 already in progress: creating tablet
W20240503 14:12:00.939123 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 567f15be6b9e4baaba9587c3634f72ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567f15be6b9e4baaba9587c3634f72ad already in progress: creating tablet
I20240503 14:12:00.939600 28653 tablet_replica.cc:330] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:00.939601 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f35e87db1c2a42a79b426b2e7767367a
I20240503 14:12:00.940152 28028 catalog_manager.cc:6554] ReplaceTablet: tablet f35e87db1c2a42a79b426b2e7767367a deleted and replaced by tablet 7de7832122804eada175f710cc5bab5e
I20240503 14:12:00.940480 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet f35e87db1c2a42a79b426b2e7767367a with delete_type TABLET_DATA_DELETED (replaced by tablet 7de7832122804eada175f710cc5bab5e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.941110 28653 raft_consensus.cc:2229] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:00.941170 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet f35e87db1c2a42a79b426b2e7767367a with delete_type TABLET_DATA_DELETED (replaced by tablet 7de7832122804eada175f710cc5bab5e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:00.941336 28653 raft_consensus.cc:2258] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.941366 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet f35e87db1c2a42a79b426b2e7767367a with delete_type TABLET_DATA_DELETED (replaced by tablet 7de7832122804eada175f710cc5bab5e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.941862 28600 tablet_bootstrap.cc:492] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.943367 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6270d949a6654e54b3cc252a54943112" } modify_external_catalogs: true
I20240503 14:12:00.943641 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6270d949a6654e54b3cc252a54943112" } modify_external_catalogs: true
I20240503 14:12:00.943714 28600 tablet_bootstrap.cc:654] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.948302 28661 ts_tablet_manager.cc:1888] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:00.948799 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f3298a69afb440bc9a9c865eb05e070d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.949554 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3298a69afb440bc9a9c865eb05e070d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.949877 28661 log.cc:1196] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e20b99c022ea437181f8d3221b952932
I20240503 14:12:00.950388 28661 ts_tablet_manager.cc:1909] T e20b99c022ea437181f8d3221b952932 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:00.951081 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ba629915ef44e0882beee2612e285c1 consensus state has no leader
I20240503 14:12:00.951567 28614 tablet_bootstrap.cc:492] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.951776 28614 ts_tablet_manager.cc:1367] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.005s
I20240503 14:12:00.952760 28614 raft_consensus.cc:348] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.952958 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 87ae10db451b4e3a8a85d1d987c447d9
I20240503 14:12:00.953068 28614 raft_consensus.cc:374] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.953142 28614 raft_consensus.cc:729] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.953469 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 87ae10db451b4e3a8a85d1d987c447d9 deleted and replaced by tablet bacf2d1091ee458c961c691093ea65a9
I20240503 14:12:00.953428 28614 consensus_queue.cc:260] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.954001 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5a5d61e70c514fbfac570c0d594e7010"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.954404 28661 tablet_replica.cc:330] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:00.954608 28661 raft_consensus.cc:2229] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:00.954664 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a5d61e70c514fbfac570c0d594e7010 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.954809 28661 raft_consensus.cc:2258] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:00.955260 28614 ts_tablet_manager.cc:1398] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:00.955657 28614 tablet_bootstrap.cc:492] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.956539 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9c60ef813e134942b2bf7a67b5f6ecf4" } modify_external_catalogs: true
I20240503 14:12:00.956789 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9c60ef813e134942b2bf7a67b5f6ecf4" } modify_external_catalogs: true
I20240503 14:12:00.957509 28614 tablet_bootstrap.cc:654] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.957613 28661 ts_tablet_manager.cc:1875] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:00.957870 28600 tablet_bootstrap.cc:492] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.958063 28600 ts_tablet_manager.cc:1367] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20240503 14:12:00.959863 28600 raft_consensus.cc:348] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.960217 28600 raft_consensus.cc:374] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.960318 28600 raft_consensus.cc:729] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.961148 28030 catalog_manager.cc:5484] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:00.961730 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 3826594d300342c7a59e0eee989a873f (table default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b]). Expected version 3 got 0
I20240503 14:12:00.962152 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_fbe6764ce0f84db08b3d9ef65525d733 [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 3 got 0
I20240503 14:12:00.962375 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 4c6782582fb543f4b621569343995dfc (table default.Table_0f8086df11104655afdf191191957937 [id=9f62dbc5baa94df18b25548d731ddeee]). Expected version 1 got 0
I20240503 14:12:00.962440 28600 consensus_queue.cc:260] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.965179 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ee72bcd4654d06a91ccfb7c24869da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.965962 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.966640 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet fd98387c1cae45dcb29c2116b6141708 with delete_type TABLET_DATA_DELETED (replaced by tablet a2bfdcb8e2b241ca9a6922fd8491e27b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.967283 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet e20b99c022ea437181f8d3221b952932 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.967926 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d0a58592c946bfa57a5698add39985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:11:59 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.968550 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet d37413e13809448ea8c94ae0e9efc618 with delete_type TABLET_DATA_DELETED (replaced by tablet d3d6e732eceb412b931c6387fb0c62bd) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.969271 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 with delete_type TABLET_DATA_DELETED (replaced by tablet bacf2d1091ee458c961c691093ea65a9) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:00.969386 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 with delete_type TABLET_DATA_DELETED (replaced by tablet bacf2d1091ee458c961c691093ea65a9) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:00.970716 28600 ts_tablet_manager.cc:1398] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:00.972157 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 with delete_type TABLET_DATA_DELETED (replaced by tablet bacf2d1091ee458c961c691093ea65a9) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:00.972795 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f35e87db1c2a42a79b426b2e7767367a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f35e87db1c2a42a79b426b2e7767367a already in progress: creating tablet
I20240503 14:12:00.973094 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e20b99c022ea437181f8d3221b952932 (table default.Table_c75bd5158ad74ccd9cc5ffd60749907a [id=d2b505b742e64fa6948bf41c834d8902]) successfully deleted
I20240503 14:12:00.974578 28600 tablet_bootstrap.cc:492] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.976987 28600 tablet_bootstrap.cc:654] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.980072 28600 tablet_bootstrap.cc:492] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:00.980170 28614 tablet_bootstrap.cc:492] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.980332 28614 ts_tablet_manager.cc:1367] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.015s
I20240503 14:12:00.980353 28600 ts_tablet_manager.cc:1367] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:12:00.981211 28600 raft_consensus.cc:348] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.981547 28600 raft_consensus.cc:374] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.981653 28600 raft_consensus.cc:729] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.981943 28600 consensus_queue.cc:260] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.982578 28600 ts_tablet_manager.cc:1398] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:00.982949 28600 tablet_bootstrap.cc:492] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:00.983832 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6106d52df3c54c2bb4abadd8605cd912
I20240503 14:12:00.984838 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9289210d002542b78ccb48780cf16018" } modify_external_catalogs: true
I20240503 14:12:00.985055 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9289210d002542b78ccb48780cf16018" } modify_external_catalogs: true
I20240503 14:12:00.985301 28600 tablet_bootstrap.cc:654] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.986179 28661 ts_tablet_manager.cc:1888] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:00.986335 28661 log.cc:1196] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/90ee72bcd4654d06a91ccfb7c24869da
I20240503 14:12:00.986735 28661 ts_tablet_manager.cc:1909] T 90ee72bcd4654d06a91ccfb7c24869da P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:00.988132 28614 raft_consensus.cc:348] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.988157 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_45bc934b7dba4fbc966160c359918f67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.988458 28614 raft_consensus.cc:374] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:00.988569 28614 raft_consensus.cc:729] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
W20240503 14:12:00.988878 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45bc934b7dba4fbc966160c359918f67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.988852 28614 consensus_queue.cc:260] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.990306 28494 raft_consensus.cc:1261] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Refusing update from remote peer a64bc15d65d84ad5a005960a9a84df9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:00.990893 28614 ts_tablet_manager.cc:1398] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:12:00.991302 28614 tablet_bootstrap.cc:492] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:00.991441 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b91808e3b9834db09a702287b58a22d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:00.991519 28651 consensus_queue.cc:1035] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:00.992132 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b91808e3b9834db09a702287b58a22d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.993441 28614 tablet_bootstrap.cc:654] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:00.994459 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 6aa994dfa54a47408bfb34bcfda9a415 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:00.995563 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_adb8e4f58ba042f0a57523a25228ca92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:00.996217 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adb8e4f58ba042f0a57523a25228ca92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:00.996856 28614 tablet_bootstrap.cc:492] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:00.997062 28614 ts_tablet_manager.cc:1367] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 14:12:00.997997 28614 raft_consensus.cc:348] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:00.998288 28614 raft_consensus.cc:374] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.000015 28614 raft_consensus.cc:729] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:00.999931 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3c2d3cac1d8c4aa4b9730b7f0baa45a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.000459 28614 consensus_queue.cc:260] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:01.000640 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c2d3cac1d8c4aa4b9730b7f0baa45a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.003392 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.004263 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.004761 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 90ee72bcd4654d06a91ccfb7c24869da (table default.Table_bc04ce776cfb4771977f33fa236ea77f [id=a9b7085b3a83482a91e6a8c888956a4e]) successfully deleted
I20240503 14:12:01.004885 28614 ts_tablet_manager.cc:1398] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
W20240503 14:12:01.005059 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 87ae10db451b4e3a8a85d1d987c447d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87ae10db451b4e3a8a85d1d987c447d9 already in progress: creating tablet
I20240503 14:12:01.006175 28614 tablet_bootstrap.cc:492] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.006364 28653 ts_tablet_manager.cc:1875] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.008126 28614 tablet_bootstrap.cc:654] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.010665 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:01.011541 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6106d52df3c54c2bb4abadd8605cd912 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6106d52df3c54c2bb4abadd8605cd912 already in progress: creating tablet
I20240503 14:12:01.013337 28653 ts_tablet_manager.cc:1888] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.013545 28653 log.cc:1196] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a50237f7cc324e61a27ef6fe2dd7f071
I20240503 14:12:01.013976 28653 ts_tablet_manager.cc:1909] T a50237f7cc324e61a27ef6fe2dd7f071 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.014973 28614 tablet_bootstrap.cc:492] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.015144 28614 ts_tablet_manager.cc:1367] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240503 14:12:01.015187 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a50237f7cc324e61a27ef6fe2dd7f071 (table default.Table_0515e83b3d914915a67f1d30f123eb09 [id=f699bab9cb6a4364a6b371df69e16222]) successfully deleted
I20240503 14:12:01.015650 28653 tablet_replica.cc:330] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.015843 28653 raft_consensus.cc:2229] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.016345 28614 raft_consensus.cc:348] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.016583 28614 raft_consensus.cc:374] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.016688 28614 raft_consensus.cc:729] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.016958 28600 tablet_bootstrap.cc:492] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.016957 28614 consensus_queue.cc:260] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.017127 28600 ts_tablet_manager.cc:1367] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.021s
I20240503 14:12:01.018092 28604 tablet_bootstrap.cc:492] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.017951 28600 raft_consensus.cc:348] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.018203 28600 raft_consensus.cc:374] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.016034 28653 raft_consensus.cc:2258] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.019331 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_059ac6056dc04f58b62976bc4cb523a2" } new_table_name: "default.Table_3a2892577d0744b5927abb18464fb658" modify_external_catalogs: true
I20240503 14:12:01.019613 28653 ts_tablet_manager.cc:1875] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.020473 28614 ts_tablet_manager.cc:1398] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:01.018316 28600 raft_consensus.cc:729] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.021577 28600 consensus_queue.cc:260] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.021859 28614 tablet_bootstrap.cc:492] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.023672 28614 tablet_bootstrap.cc:654] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.024314 28604 ts_tablet_manager.cc:1367] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.005s
I20240503 14:12:01.027803 28600 ts_tablet_manager.cc:1398] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:12:01.028216 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet fdccb370198b4eb69768aeed787fd682
I20240503 14:12:01.029220 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b2d7140d9a16416a97ef0b0b9b506468" } new_table_name: "default.Table_456be1a488a44191be1e21de223834c9" modify_external_catalogs: true
I20240503 14:12:01.031131 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet fdccb370198b4eb69768aeed787fd682 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.031659 28600 tablet_bootstrap.cc:492] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.031898 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet fdccb370198b4eb69768aeed787fd682 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.031937 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet fdccb370198b4eb69768aeed787fd682 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.031618 28604 raft_consensus.cc:348] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.034077 28604 raft_consensus.cc:374] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.034202 28604 raft_consensus.cc:729] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.036280 28600 tablet_bootstrap.cc:654] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.036468 28603 tablet_replica.cc:330] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.036752 28603 raft_consensus.cc:2229] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.037084 28603 pending_rounds.cc:62] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:01.037173 28603 pending_rounds.cc:68] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597653009076224 op_type: NO_OP noop_request { }
I20240503 14:12:01.037304 28603 raft_consensus.cc:2866] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:01.037427 28603 raft_consensus.cc:2258] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.037596 28653 ts_tablet_manager.cc:1888] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.037748 28653 log.cc:1196] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f35e87db1c2a42a79b426b2e7767367a
I20240503 14:12:01.038137 28653 ts_tablet_manager.cc:1909] T f35e87db1c2a42a79b426b2e7767367a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.039271 28653 tablet_replica.cc:330] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.039500 28653 raft_consensus.cc:2229] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.039531 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_50ca74d27bab49ff9f56b8e584929e8a" } new_table_name: "default.Table_7d8e4d1ecd794871b9d73289f8f43c9a" modify_external_catalogs: true
I20240503 14:12:01.039688 28653 raft_consensus.cc:2258] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.039563 28604 consensus_queue.cc:260] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.039852 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet f35e87db1c2a42a79b426b2e7767367a (table default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]) successfully deleted
I20240503 14:12:01.040385 28653 ts_tablet_manager.cc:1875] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.041296 28604 ts_tablet_manager.cc:1398] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240503 14:12:01.041640 28604 tablet_bootstrap.cc:492] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.042443 28614 tablet_bootstrap.cc:492] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.042670 28614 ts_tablet_manager.cc:1367] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.006s
I20240503 14:12:01.043442 28604 tablet_bootstrap.cc:654] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.043529 28614 raft_consensus.cc:348] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.043800 28614 raft_consensus.cc:374] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.043869 28614 raft_consensus.cc:729] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.044178 28614 consensus_queue.cc:260] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.045562 28357 raft_consensus.cc:1261] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Refusing update from remote peer a64bc15d65d84ad5a005960a9a84df9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:01.046033 28614 ts_tablet_manager.cc:1398] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:01.046602 28614 tablet_bootstrap.cc:492] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.048520 28614 tablet_bootstrap.cc:654] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.048913 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.049533 28662 raft_consensus.cc:482] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.049751 28662 raft_consensus.cc:504] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.050052 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3826594d300342c7a59e0eee989a873f (table default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b]). Expected version 3 got 0
I20240503 14:12:01.050311 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_fbe6764ce0f84db08b3d9ef65525d733 [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 3 got 0
I20240503 14:12:01.050560 28662 leader_election.cc:288] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.051944 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 3826594d300342c7a59e0eee989a873f (table default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b]). Expected version 3 got 0
I20240503 14:12:01.052284 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35e87db1c2a42a79b426b2e7767367a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.053997 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_25378c7373494eb6a7e7e3bebd175e6b" } new_table_name: "default.Table_55fe0191399e49d19d8ffd2e6cd0a77c" modify_external_catalogs: true
W20240503 14:12:01.054253 28290 leader_election.cc:341] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: f35e87db1c2a42a79b426b2e7767367a
I20240503 14:12:01.054685 28687 mvcc.cc:205] Tried to move back new op lower bound from 7023597653944205312 to 7023597652403023872. Current Snapshot: MvccSnapshot[applied={T|T < 7023597653944205312}]
I20240503 14:12:01.058043 28661 tablet_replica.cc:330] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:01.057922 28692 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:01.058333 28661 raft_consensus.cc:2229] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.058557 28661 raft_consensus.cc:2258] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:01.062126 28290 leader_election.cc:341] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.062829 28610 consensus_queue.cc:1035] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:01.063551 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 67ec68f75fa2435698383a03874a5bae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67ec68f75fa2435698383a03874a5bae already in progress: creating tablet
I20240503 14:12:01.063764 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet d37413e13809448ea8c94ae0e9efc618 with delete_type TABLET_DATA_DELETED (replaced by tablet d3d6e732eceb412b931c6387fb0c62bd) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:01.063906 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6aa994dfa54a47408bfb34bcfda9a415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6aa994dfa54a47408bfb34bcfda9a415 already in progress: creating tablet
W20240503 14:12:01.064172 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 567f15be6b9e4baaba9587c3634f72ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567f15be6b9e4baaba9587c3634f72ad already in progress: creating tablet
W20240503 14:12:01.064400 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a50237f7cc324e61a27ef6fe2dd7f071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a50237f7cc324e61a27ef6fe2dd7f071 already in progress: creating tablet
W20240503 14:12:01.064623 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f35e87db1c2a42a79b426b2e7767367a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f35e87db1c2a42a79b426b2e7767367a already in progress: creating tablet
W20240503 14:12:01.064824 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 87ae10db451b4e3a8a85d1d987c447d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87ae10db451b4e3a8a85d1d987c447d9 already in progress: creating tablet
W20240503 14:12:01.065047 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6106d52df3c54c2bb4abadd8605cd912 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6106d52df3c54c2bb4abadd8605cd912 already in progress: creating tablet
I20240503 14:12:01.066227 28290 leader_election.cc:302] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.066619 28662 raft_consensus.cc:2726] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.069085 28600 tablet_bootstrap.cc:492] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.069317 28600 ts_tablet_manager.cc:1367] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.010s
I20240503 14:12:01.070255 28600 raft_consensus.cc:348] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.070557 28600 raft_consensus.cc:374] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.070636 28600 raft_consensus.cc:729] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.071009 28600 consensus_queue.cc:260] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.073475 28603 ts_tablet_manager.cc:1875] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.076846 28600 ts_tablet_manager.cc:1398] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 14:12:01.079931 28653 ts_tablet_manager.cc:1888] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.080217 28653 log.cc:1196] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/90ee72bcd4654d06a91ccfb7c24869da
I20240503 14:12:01.080649 28653 ts_tablet_manager.cc:1909] T 90ee72bcd4654d06a91ccfb7c24869da P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.082095 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 90ee72bcd4654d06a91ccfb7c24869da (table default.Table_bc04ce776cfb4771977f33fa236ea77f [id=a9b7085b3a83482a91e6a8c888956a4e]) successfully deleted
I20240503 14:12:01.084527 28604 tablet_bootstrap.cc:492] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.085521 28604 ts_tablet_manager.cc:1367] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.002s
I20240503 14:12:01.087570 28600 tablet_bootstrap.cc:492] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.087627 28614 tablet_bootstrap.cc:492] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.087795 28614 ts_tablet_manager.cc:1367] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.013s
I20240503 14:12:01.088830 28614 raft_consensus.cc:348] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.089115 28614 raft_consensus.cc:374] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.089216 28614 raft_consensus.cc:729] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.089581 28600 tablet_bootstrap.cc:654] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.089581 28614 consensus_queue.cc:260] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.089920 28604 raft_consensus.cc:348] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.091073 28604 raft_consensus.cc:374] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.091532 28604 raft_consensus.cc:729] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:01.097757 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 71598d0a44384a3b93dec30d6c60ecd5 because the tablet was not found. No further retry: Not found: Tablet not found: 71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:01.098079 28653 tablet_replica.cc:330] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:01.098171 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet fd98387c1cae45dcb29c2116b6141708 because the tablet was not found. No further retry: Not found: Tablet not found: fd98387c1cae45dcb29c2116b6141708
I20240503 14:12:01.098304 28653 raft_consensus.cc:2229] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.098511 28653 raft_consensus.cc:2258] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.099107 28653 ts_tablet_manager.cc:1875] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.102300 28604 consensus_queue.cc:260] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.105530 28614 ts_tablet_manager.cc:1398] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20240503 14:12:01.107159 28614 tablet_bootstrap.cc:492] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.109197 28614 tablet_bootstrap.cc:654] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.110785 28692 tablet.cc:1717] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:01.111675 28610 raft_consensus.cc:482] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.111874 28610 raft_consensus.cc:504] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.112600 28610 leader_election.cc:288] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.113994 28683 mvcc.cc:205] Tried to move back new op lower bound from 7023597654423441408 to 7023597652403023872. Current Snapshot: MvccSnapshot[applied={T|T < 7023597654423441408}]
I20240503 14:12:01.114470 28622 raft_consensus.cc:482] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.114703 28622 raft_consensus.cc:504] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.115512 28622 leader_election.cc:288] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:01.115588 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f54dc481079c4a5db5b54d485991fa65" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:01.115903 28356 raft_consensus.cc:2454] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:01.116425 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71598d0a44384a3b93dec30d6c60ecd5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.116708 28493 raft_consensus.cc:2454] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:01.116873 28290 leader_election.cc:341] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:01.117182 28290 leader_election.cc:302] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:01.117715 28622 raft_consensus.cc:2781] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.117703 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f54dc481079c4a5db5b54d485991fa65" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.117825 28622 raft_consensus.cc:482] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.117936 28622 raft_consensus.cc:3037] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.118502 28492 raft_consensus.cc:3037] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.118894 28150 leader_election.cc:302] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:01.118474 28355 raft_consensus.cc:1261] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Refusing update from remote peer a64bc15d65d84ad5a005960a9a84df9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:01.119655 28610 raft_consensus.cc:2781] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.119764 28610 raft_consensus.cc:482] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.119858 28610 raft_consensus.cc:3037] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.121007 28622 raft_consensus.cc:504] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.121420 28604 ts_tablet_manager.cc:1398] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.035s user 0.004s sys 0.000s
I20240503 14:12:01.121774 28492 raft_consensus.cc:1261] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Refusing update from remote peer a64bc15d65d84ad5a005960a9a84df9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:01.122428 28604 tablet_bootstrap.cc:492] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.122639 28610 raft_consensus.cc:504] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:01.122861 28290 leader_election.cc:341] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:01.123442 28610 leader_election.cc:288] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.123790 28622 leader_election.cc:288] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:01.123979 28150 leader_election.cc:341] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.124519 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f54dc481079c4a5db5b54d485991fa65" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.124480 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71598d0a44384a3b93dec30d6c60ecd5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.124938 28492 raft_consensus.cc:3037] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.125174 28651 consensus_queue.cc:1035] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:01.126202 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f54dc481079c4a5db5b54d485991fa65" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:01.126472 28355 raft_consensus.cc:3037] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.128034 28492 raft_consensus.cc:2454] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:01.128832 28290 leader_election.cc:302] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:01.129444 28622 raft_consensus.cc:2781] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.130527 28604 tablet_bootstrap.cc:654] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.130522 28355 raft_consensus.cc:2454] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:01.131623 28622 raft_consensus.cc:686] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:01.132051 28622 consensus_queue.cc:237] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.094687 28603 ts_tablet_manager.cc:1888] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:01.096827 28661 ts_tablet_manager.cc:1875] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.136915 28610 raft_consensus.cc:482] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:01.137202 28150 leader_election.cc:341] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.137742 28150 leader_election.cc:302] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.138046 28603 log.cc:1196] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/fdccb370198b4eb69768aeed787fd682
I20240503 14:12:01.138214 28637 consensus_queue.cc:1035] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:01.138900 28637 raft_consensus.cc:2781] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.139055 28637 raft_consensus.cc:686] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:01.139377 28637 consensus_queue.cc:237] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.144979 28603 ts_tablet_manager.cc:1909] T fdccb370198b4eb69768aeed787fd682 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.146132 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet fdccb370198b4eb69768aeed787fd682 (table default.Table_9c60ef813e134942b2bf7a67b5f6ecf4 [id=9f10df997dc04f05a4f052f20c4a2fc5]) successfully deleted
I20240503 14:12:01.147747 28651 raft_consensus.cc:482] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.147958 28651 raft_consensus.cc:504] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.148751 28651 leader_election.cc:288] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.149155 28610 raft_consensus.cc:504] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.149899 28610 leader_election.cc:288] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:01.150781 28653 ts_tablet_manager.cc:1888] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.151224 28653 log.cc:1196] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e20b99c022ea437181f8d3221b952932
I20240503 14:12:01.152175 28600 tablet_bootstrap.cc:492] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.152562 28600 ts_tablet_manager.cc:1367] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.000s
I20240503 14:12:01.153550 28600 raft_consensus.cc:348] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.153777 28600 raft_consensus.cc:374] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.153865 28600 raft_consensus.cc:729] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.154187 28600 consensus_queue.cc:260] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.154814 28600 ts_tablet_manager.cc:1398] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:01.155164 28600 tablet_bootstrap.cc:492] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.159157 28614 tablet_bootstrap.cc:492] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.159384 28614 ts_tablet_manager.cc:1367] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.010s
I20240503 14:12:01.131304 28693 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:01.160317 28614 raft_consensus.cc:348] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.160619 28614 raft_consensus.cc:374] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.160748 28614 raft_consensus.cc:729] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.161118 28614 consensus_queue.cc:260] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.161811 28614 ts_tablet_manager.cc:1398] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:01.162276 28614 tablet_bootstrap.cc:492] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.164471 28614 tablet_bootstrap.cc:654] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.170800 28600 tablet_bootstrap.cc:654] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.174818 28653 ts_tablet_manager.cc:1909] T e20b99c022ea437181f8d3221b952932 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.176321 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e20b99c022ea437181f8d3221b952932 (table default.Table_c75bd5158ad74ccd9cc5ffd60749907a [id=d2b505b742e64fa6948bf41c834d8902]) successfully deleted
I20240503 14:12:01.178143 28705 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:01.179831 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa994dfa54a47408bfb34bcfda9a415" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.181533 28692 tablet.cc:1717] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:01.185333 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 67ec68f75fa2435698383a03874a5bae from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:01.186105 28150 leader_election.cc:341] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:01.186367 28150 leader_election.cc:341] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 42d0a58592c946bfa57a5698add39985
W20240503 14:12:01.187201 28150 leader_election.cc:341] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 42d0a58592c946bfa57a5698add39985
I20240503 14:12:01.187338 28150 leader_election.cc:302] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.187592 28637 raft_consensus.cc:2726] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.190373 28653 tablet_replica.cc:330] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.190658 28653 raft_consensus.cc:2229] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.190855 28653 raft_consensus.cc:2258] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.191619 28653 ts_tablet_manager.cc:1875] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:01.193634 28704 log.cc:924] Time spent T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240503 14:12:01.198415 28604 tablet_bootstrap.cc:492] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.198611 28604 ts_tablet_manager.cc:1367] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.008s
I20240503 14:12:01.199486 28604 raft_consensus.cc:348] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.199733 28604 raft_consensus.cc:374] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.199826 28604 raft_consensus.cc:729] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.200235 28604 consensus_queue.cc:260] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.201004 28604 ts_tablet_manager.cc:1398] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:01.201732 28604 tablet_bootstrap.cc:492] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.203758 28693 tablet.cc:1717] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:01.206092 28600 tablet_bootstrap.cc:492] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.206341 28600 ts_tablet_manager.cc:1367] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.003s
I20240503 14:12:01.206648 28604 tablet_bootstrap.cc:654] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.181207 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa994dfa54a47408bfb34bcfda9a415" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:01.207434 28600 raft_consensus.cc:348] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.207705 28600 raft_consensus.cc:374] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.207777 28600 raft_consensus.cc:729] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.207811 28355 raft_consensus.cc:2454] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:01.194387 28662 raft_consensus.cc:482] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.208241 28662 raft_consensus.cc:504] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.207507 28661 ts_tablet_manager.cc:1888] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.208613 28661 log.cc:1196] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/fdccb370198b4eb69768aeed787fd682
I20240503 14:12:01.208551 28600 consensus_queue.cc:260] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.209084 28661 ts_tablet_manager.cc:1909] T fdccb370198b4eb69768aeed787fd682 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:01.209077 28662 leader_election.cc:288] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.209671 28150 leader_election.cc:302] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.209738 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ec68f75fa2435698383a03874a5bae" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.210022 28492 raft_consensus.cc:2454] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:01.210215 28637 raft_consensus.cc:2781] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.210368 28637 raft_consensus.cc:482] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.205895 28696 mvcc.cc:205] Tried to move back new op lower bound from 7023597653944205312 to 7023597652403023872. Current Snapshot: MvccSnapshot[applied={T|T < 7023597653944205312}]
I20240503 14:12:01.210476 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ec68f75fa2435698383a03874a5bae" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:01.210747 28217 raft_consensus.cc:2454] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:01.212504 28711 tablet.cc:1717] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:01.212479 28637 raft_consensus.cc:3037] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.212747 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_36901c8428f346ae8117eb126334063b" } modify_external_catalogs: true
I20240503 14:12:01.212990 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_36901c8428f346ae8117eb126334063b" } modify_external_catalogs: true
I20240503 14:12:01.213733 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4033322e04a3488cbc8be0fe6a61bec1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.214336 28661 tablet_replica.cc:330] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:01.214442 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4033322e04a3488cbc8be0fe6a61bec1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.214550 28661 raft_consensus.cc:2229] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.214756 28661 raft_consensus.cc:2258] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.215711 28637 raft_consensus.cc:504] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.216387 28662 raft_consensus.cc:482] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.216605 28662 raft_consensus.cc:504] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.217454 28662 leader_election.cc:288] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.217631 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet fdccb370198b4eb69768aeed787fd682 (table default.Table_9c60ef813e134942b2bf7a67b5f6ecf4 [id=9f10df997dc04f05a4f052f20c4a2fc5]) successfully deleted
I20240503 14:12:01.218981 28604 tablet_bootstrap.cc:492] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.218911 28290 leader_election.cc:302] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:01.219194 28604 ts_tablet_manager.cc:1367] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240503 14:12:01.220119 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c0611cf7aa4ff3b4e7e6081558729b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.220175 28604 raft_consensus.cc:348] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.220431 28604 raft_consensus.cc:374] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.220516 28604 raft_consensus.cc:729] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.220809 28604 consensus_queue.cc:260] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.221304 28662 raft_consensus.cc:2781] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.221449 28662 raft_consensus.cc:482] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.221546 28662 raft_consensus.cc:3037] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.225538 28662 raft_consensus.cc:504] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.226562 28662 leader_election.cc:288] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.227491 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ec68f75fa2435698383a03874a5bae" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.227778 28492 raft_consensus.cc:3037] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.230943 28492 raft_consensus.cc:2454] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
W20240503 14:12:01.231616 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b166fe6366ba496a869af1b106fe9d8e consensus state has no leader
I20240503 14:12:01.232159 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0f8086df11104655afdf191191957937" } modify_external_catalogs: true
I20240503 14:12:01.232447 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0f8086df11104655afdf191191957937" } modify_external_catalogs: true
I20240503 14:12:01.232388 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ec68f75fa2435698383a03874a5bae" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:01.232587 28604 ts_tablet_manager.cc:1398] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:12:01.232631 28217 raft_consensus.cc:3037] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.232987 28604 tablet_bootstrap.cc:492] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.234889 28604 tablet_bootstrap.cc:654] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.236359 28217 raft_consensus.cc:2454] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
W20240503 14:12:01.236539 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 559342d357d64f9eb9e3ab42ccb75942 has no consensus state
W20240503 14:12:01.237053 28290 leader_election.cc:341] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.237054 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_31bf8350c35e43409cfb20708c238df5" } new_table_name: "default.Table_8d906bfe77124e609d331427b67fabd0" modify_external_catalogs: true
I20240503 14:12:01.237319 28290 leader_election.cc:302] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:01.237692 28290 leader_election.cc:341] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 38c0611cf7aa4ff3b4e7e6081558729b
I20240503 14:12:01.237809 28290 leader_election.cc:302] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.238842 28637 leader_election.cc:288] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.239188 28662 raft_consensus.cc:2781] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.240200 28662 raft_consensus.cc:686] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:01.240584 28662 consensus_queue.cc:237] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.240841 28715 raft_consensus.cc:2726] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.243029 28661 ts_tablet_manager.cc:1875] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.244374 28600 ts_tablet_manager.cc:1398] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.038s user 0.003s sys 0.000s
I20240503 14:12:01.244877 28600 tablet_bootstrap.cc:492] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.246946 28600 tablet_bootstrap.cc:654] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.256207 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa994dfa54a47408bfb34bcfda9a415" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.256611 28492 raft_consensus.cc:3037] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.259958 28492 raft_consensus.cc:2454] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:01.261500 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa994dfa54a47408bfb34bcfda9a415" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:01.264870 28150 leader_election.cc:302] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:01.266718 28703 raft_consensus.cc:482] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.267102 28703 raft_consensus.cc:504] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.271482 28703 leader_election.cc:288] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.271951 28357 raft_consensus.cc:3037] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.274165 28030 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 4c6782582fb543f4b621569343995dfc (table default.Table_0f8086df11104655afdf191191957937 [id=9f62dbc5baa94df18b25548d731ddeee]). Expected version 1 got 0
I20240503 14:12:01.275110 28357 raft_consensus.cc:2454] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:01.275730 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 5d637087c60144aaaa9b0965110d280e (table default.Table_25378c7373494eb6a7e7e3bebd175e6b [id=500a3febdac042138ea31f6f3c8d02b5]). Expected version 1 got 0
I20240503 14:12:01.276695 28637 raft_consensus.cc:2781] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.277103 28614 tablet_bootstrap.cc:492] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.277279 28614 ts_tablet_manager.cc:1367] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.016s
I20240503 14:12:01.277395 28031 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_fbe6764ce0f84db08b3d9ef65525d733 [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 3 got 0
I20240503 14:12:01.277858 28692 tablet.cc:1722] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:01.278240 28614 raft_consensus.cc:348] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.278507 28614 raft_consensus.cc:374] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.278618 28614 raft_consensus.cc:729] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.279868 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f54dc481079c4a5db5b54d485991fa65
I20240503 14:12:01.280261 28637 raft_consensus.cc:686] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:01.280700 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet 4c6782582fb543f4b621569343995dfc from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:01.281177 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2bfdcb8e2b241ca9a6922fd8491e27b consensus state has no leader
I20240503 14:12:01.281534 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 567f15be6b9e4baaba9587c3634f72ad from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.281333 28637 consensus_queue.cc:237] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:01.283151 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d637087c60144aaaa9b0965110d280e consensus state has no leader
I20240503 14:12:01.283499 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 87ae10db451b4e3a8a85d1d987c447d9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.283274 28614 consensus_queue.cc:260] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.284658 28614 ts_tablet_manager.cc:1398] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:01.285462 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.285629 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 92c800e997e944158f7c839dd0e00d44 (DEFAULT_TABLE table=default.Table_5f10f98658754315a6cc9b1d7ffc19cb [id=d9b865dc744148df9fc1ab6e79c5043d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.285926 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2502a13e8cbc4a3abfed225f94c44d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.286154 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c800e997e944158f7c839dd0e00d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.288594 28473 tablet_service.cc:1467] Processing CreateTablet for tablet eb79edba14b646ef9be239b05a291d71 (DEFAULT_TABLE table=default.Table_3dd432e943764d9296fada2cf3c0ca47 [id=6791a22730fc49c59b2713384ee15850]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.289131 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb79edba14b646ef9be239b05a291d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.292361 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 (DEFAULT_TABLE table=default.Table_7d8e4d1ecd794871b9d73289f8f43c9a [id=c6805f8e2643425dbed90e5ebe7d3a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.292455 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ae10db451b4e3a8a85d1d987c447d9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.292984 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559342d357d64f9eb9e3ab42ccb75942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.294077 28614 tablet_bootstrap.cc:492] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.294792 28653 ts_tablet_manager.cc:1888] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.295090 28653 log.cc:1196] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/42d0a58592c946bfa57a5698add39985
I20240503 14:12:01.295562 28653 ts_tablet_manager.cc:1909] T 42d0a58592c946bfa57a5698add39985 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.296569 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.296983 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ae10db451b4e3a8a85d1d987c447d9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:01.297122 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2502a13e8cbc4a3abfed225f94c44d2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:01.299530 28150 leader_election.cc:341] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:01.299851 28150 leader_election.cc:341] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.300122 28150 leader_election.cc:302] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.299883 28719 tablet.cc:1722] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:01.300627 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 92c800e997e944158f7c839dd0e00d44 (DEFAULT_TABLE table=default.Table_5f10f98658754315a6cc9b1d7ffc19cb [id=d9b865dc744148df9fc1ab6e79c5043d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.301057 28703 raft_consensus.cc:2726] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.301225 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c800e997e944158f7c839dd0e00d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.302675 28192 tablet_service.cc:1467] Processing CreateTablet for tablet eb79edba14b646ef9be239b05a291d71 (DEFAULT_TABLE table=default.Table_3dd432e943764d9296fada2cf3c0ca47 [id=6791a22730fc49c59b2713384ee15850]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.303267 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb79edba14b646ef9be239b05a291d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.308580 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 (DEFAULT_TABLE table=default.Table_7d8e4d1ecd794871b9d73289f8f43c9a [id=c6805f8e2643425dbed90e5ebe7d3a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.309170 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559342d357d64f9eb9e3ab42ccb75942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.309023 28614 tablet_bootstrap.cc:654] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.314452 28600 tablet_bootstrap.cc:492] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.314666 28600 ts_tablet_manager.cc:1367] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.003s
I20240503 14:12:01.315594 28600 raft_consensus.cc:348] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.315853 28600 raft_consensus.cc:374] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.315956 28600 raft_consensus.cc:729] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.316730 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet f54dc481079c4a5db5b54d485991fa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.321650 28653 tablet_replica.cc:330] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.321923 28653 raft_consensus.cc:2229] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.322177 28653 raft_consensus.cc:2258] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.322580 28600 consensus_queue.cc:260] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.328126 28600 ts_tablet_manager.cc:1398] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 14:12:01.328164 28614 tablet_bootstrap.cc:492] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.328557 28614 ts_tablet_manager.cc:1367] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.004s
I20240503 14:12:01.329298 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5f10f98658754315a6cc9b1d7ffc19cb" } modify_external_catalogs: true
I20240503 14:12:01.329548 28653 ts_tablet_manager.cc:1875] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.329579 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5f10f98658754315a6cc9b1d7ffc19cb" } modify_external_catalogs: true
I20240503 14:12:01.330114 28614 raft_consensus.cc:348] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.330523 28614 raft_consensus.cc:374] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.330657 28614 raft_consensus.cc:729] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.331189 28614 consensus_queue.cc:260] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.332525 28614 ts_tablet_manager.cc:1398] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:01.332355 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_706c4da08fa84bbaa207fc71eb482682"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.333133 28614 tablet_bootstrap.cc:492] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:01.333137 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_706c4da08fa84bbaa207fc71eb482682 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.336058 28614 tablet_bootstrap.cc:654] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.336282 28672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:01.336580 28672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:01.337540 28600 tablet_bootstrap.cc:492] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.339890 28600 tablet_bootstrap.cc:654] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.343609 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet f54dc481079c4a5db5b54d485991fa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.347061 28614 tablet_bootstrap.cc:492] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.347438 28614 ts_tablet_manager.cc:1367] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240503 14:12:01.347687 28653 ts_tablet_manager.cc:1888] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.349097 28614 raft_consensus.cc:348] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.349542 28614 raft_consensus.cc:374] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.349766 28653 log.cc:1196] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/87ae10db451b4e3a8a85d1d987c447d9
I20240503 14:12:01.349735 28614 raft_consensus.cc:729] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.350241 28653 ts_tablet_manager.cc:1909] T 87ae10db451b4e3a8a85d1d987c447d9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.350292 28614 consensus_queue.cc:260] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.351277 28600 tablet_bootstrap.cc:492] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.351511 28600 ts_tablet_manager.cc:1367] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20240503 14:12:01.351570 28614 ts_tablet_manager.cc:1398] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:01.352195 28614 tablet_bootstrap.cc:492] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.353183 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 92c800e997e944158f7c839dd0e00d44 (DEFAULT_TABLE table=default.Table_5f10f98658754315a6cc9b1d7ffc19cb [id=d9b865dc744148df9fc1ab6e79c5043d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.353818 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c800e997e944158f7c839dd0e00d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.354212 28600 raft_consensus.cc:348] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.354622 28600 raft_consensus.cc:374] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.354729 28600 raft_consensus.cc:729] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.355103 28600 consensus_queue.cc:260] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.355284 28614 tablet_bootstrap.cc:654] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.360548 28600 ts_tablet_manager.cc:1398] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 14:12:01.363286 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 (DEFAULT_TABLE table=default.Table_7d8e4d1ecd794871b9d73289f8f43c9a [id=c6805f8e2643425dbed90e5ebe7d3a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.363381 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.363816 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet f54dc481079c4a5db5b54d485991fa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.364593 28603 tablet_replica.cc:330] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.364864 28603 raft_consensus.cc:2229] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.365124 28603 raft_consensus.cc:2258] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.363893 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559342d357d64f9eb9e3ab42ccb75942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.366253 28603 ts_tablet_manager.cc:1875] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.366587 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2502a13e8cbc4a3abfed225f94c44d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.373526 28600 tablet_bootstrap.cc:492] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.375718 28600 tablet_bootstrap.cc:654] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.376606 28637 raft_consensus.cc:482] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.376849 28637 raft_consensus.cc:504] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.377738 28637 leader_election.cc:288] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.378508 28653 tablet_replica.cc:330] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.378778 28653 raft_consensus.cc:2229] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.378968 28653 raft_consensus.cc:2258] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.379478 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6106d52df3c54c2bb4abadd8605cd912" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.380024 28653 ts_tablet_manager.cc:1875] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:01.380508 28150 leader_election.cc:341] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.381062 28622 raft_consensus.cc:482] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.381253 28622 raft_consensus.cc:504] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.381985 28622 leader_election.cc:288] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.268067 28705 tablet.cc:1717] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:01.383740 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6782582fb543f4b621569343995dfc" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.384678 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6106d52df3c54c2bb4abadd8605cd912" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:01.385753 28150 leader_election.cc:341] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.385928 28150 leader_election.cc:302] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:01.386171 28150 raft_consensus.cc:2670] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:01.387081 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6782582fb543f4b621569343995dfc" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:01.387387 28217 raft_consensus.cc:2454] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:01.389792 28290 leader_election.cc:341] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.390259 28614 tablet_bootstrap.cc:492] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.392513 28614 ts_tablet_manager.cc:1367] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.009s
I20240503 14:12:01.392522 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 42d0a58592c946bfa57a5698add39985 (table default.Table_0a89e43de47c4ee2ba100e7dbf451d84 [id=72babfde5906464b9a4e55e85c5db68e]) successfully deleted
W20240503 14:12:01.393054 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d37413e13809448ea8c94ae0e9efc618 because the tablet was not found. No further retry: Not found: Tablet not found: d37413e13809448ea8c94ae0e9efc618
I20240503 14:12:01.393571 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 87ae10db451b4e3a8a85d1d987c447d9 (table default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]) successfully deleted
I20240503 14:12:01.390187 28290 leader_election.cc:302] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.394768 28622 raft_consensus.cc:2781] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.394923 28622 raft_consensus.cc:482] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.395010 28622 raft_consensus.cc:3037] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.395998 28653 ts_tablet_manager.cc:1888] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.396210 28653 log.cc:1196] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6106d52df3c54c2bb4abadd8605cd912
I20240503 14:12:01.396631 28653 ts_tablet_manager.cc:1909] T 6106d52df3c54c2bb4abadd8605cd912 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.396581 28614 raft_consensus.cc:348] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.397246 28614 raft_consensus.cc:374] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.397524 28614 raft_consensus.cc:729] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.399400 28333 tablet_service.cc:1467] Processing CreateTablet for tablet eb79edba14b646ef9be239b05a291d71 (DEFAULT_TABLE table=default.Table_3dd432e943764d9296fada2cf3c0ca47 [id=6791a22730fc49c59b2713384ee15850]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.400213 28604 tablet_bootstrap.cc:492] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.400416 28604 ts_tablet_manager.cc:1367] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.168s user 0.039s sys 0.036s
I20240503 14:12:01.401440 28600 tablet_bootstrap.cc:492] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.401738 28600 ts_tablet_manager.cc:1367] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.004s
I20240503 14:12:01.402549 28600 raft_consensus.cc:348] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.402757 28600 raft_consensus.cc:374] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.402856 28600 raft_consensus.cc:729] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.403096 28600 consensus_queue.cc:260] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.403512 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb79edba14b646ef9be239b05a291d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.401564 28604 raft_consensus.cc:348] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.403653 28604 raft_consensus.cc:374] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.403782 28604 raft_consensus.cc:729] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.403676 28603 ts_tablet_manager.cc:1888] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.404055 28604 consensus_queue.cc:260] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.404928 28705 tablet.cc:1717] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:01.405645 28604 ts_tablet_manager.cc:1398] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:12:01.407179 28711 tablet.cc:1722] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:01.405105 28603 log.cc:1196] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/f54dc481079c4a5db5b54d485991fa65
I20240503 14:12:01.407876 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6106d52df3c54c2bb4abadd8605cd912 (table default.Table_6270d949a6654e54b3cc252a54943112 [id=ada8758d8af14ac7bc31969934efb7d1]) successfully deleted
I20240503 14:12:01.408246 28603 ts_tablet_manager.cc:1909] T f54dc481079c4a5db5b54d485991fa65 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.408363 28622 raft_consensus.cc:504] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.409139 28622 leader_election.cc:288] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.409557 28604 tablet_bootstrap.cc:492] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.408042 28614 consensus_queue.cc:260] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.411135 28614 ts_tablet_manager.cc:1398] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 14:12:01.411204 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6782582fb543f4b621569343995dfc" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:01.411177 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_94abbca6df6f456fb210dd71f21384af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.411463 28217 raft_consensus.cc:3037] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:01.411901 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94abbca6df6f456fb210dd71f21384af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.412354 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6782582fb543f4b621569343995dfc" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.415158 28217 raft_consensus.cc:2454] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:01.417953 28604 tablet_bootstrap.cc:654] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:01.418434 28290 leader_election.cc:341] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.418782 28290 leader_election.cc:302] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.419389 28622 raft_consensus.cc:2781] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.419776 28622 raft_consensus.cc:686] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:01.420269 28622 consensus_queue.cc:237] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.420822 28661 ts_tablet_manager.cc:1888] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.421921 28614 tablet_bootstrap.cc:492] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.423369 28661 log.cc:1196] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/d37413e13809448ea8c94ae0e9efc618
I20240503 14:12:01.425230 28661 ts_tablet_manager.cc:1909] T d37413e13809448ea8c94ae0e9efc618 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:01.425333 28600 ts_tablet_manager.cc:1398] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240503 14:12:01.426601 28614 tablet_bootstrap.cc:654] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.429488 28600 tablet_bootstrap.cc:492] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.431615 28600 tablet_bootstrap.cc:654] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.432497 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet f54dc481079c4a5db5b54d485991fa65 (table default.Table_9289210d002542b78ccb48780cf16018 [id=3e4bfd54a342491b86e978c08faa5c8b]) successfully deleted
I20240503 14:12:01.436142 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet d37413e13809448ea8c94ae0e9efc618 (table default.Table_4a16c898f66d4ad999025524253f572b [id=ebd0741cd2a143698fd3cea6cf80b853]) successfully deleted
I20240503 14:12:01.437050 28653 tablet_replica.cc:330] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.437335 28653 raft_consensus.cc:2229] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.437548 28653 raft_consensus.cc:2258] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.438228 28653 ts_tablet_manager.cc:1875] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:01.439473 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc337647c7cd402e91720478380f4696 has no consensus state
I20240503 14:12:01.440213 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_22032e3385624e98a1b0bb1484f51c64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.441020 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22032e3385624e98a1b0bb1484f51c64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.442598 28027 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_fbe6764ce0f84db08b3d9ef65525d733 [id=3a24797cc53b439da44cc77025fa9f92] alter complete (version 3)
I20240503 14:12:01.443838 28614 tablet_bootstrap.cc:492] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.444135 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_baa3d7860d0b469899a33511f6bb9a29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.444025 28614 ts_tablet_manager.cc:1367] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
W20240503 14:12:01.444883 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baa3d7860d0b469899a33511f6bb9a29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.445420 28614 raft_consensus.cc:348] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.445847 28622 raft_consensus.cc:482] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.446025 28622 raft_consensus.cc:504] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.446534 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f9713865e09349ce9f9ae1d4297da2d0
I20240503 14:12:01.446782 28622 leader_election.cc:288] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.447563 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4dd04c06a7a04769b9b6eef8faac57a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.448364 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dd04c06a7a04769b9b6eef8faac57a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.448418 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d637087c60144aaaa9b0965110d280e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.448581 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d637087c60144aaaa9b0965110d280e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:01.448834 28217 raft_consensus.cc:2454] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:01.449200 28290 leader_election.cc:341] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.449321 28622 raft_consensus.cc:482] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.449445 28600 tablet_bootstrap.cc:492] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.449514 28290 leader_election.cc:302] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.449631 28600 ts_tablet_manager.cc:1367] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 14:12:01.449503 28622 raft_consensus.cc:504] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.450132 28715 raft_consensus.cc:2781] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.450274 28715 raft_consensus.cc:482] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.450330 28622 leader_election.cc:288] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:01.450389 28715 raft_consensus.cc:3037] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.450745 28600 raft_consensus.cc:348] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.450987 28600 raft_consensus.cc:374] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.451063 28600 raft_consensus.cc:729] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.451413 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b91808e3b9834db09a702287b58a22d4" } modify_external_catalogs: true
I20240503 14:12:01.451403 28600 consensus_queue.cc:260] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.451638 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b91808e3b9834db09a702287b58a22d4" } modify_external_catalogs: true
I20240503 14:12:01.453353 28600 ts_tablet_manager.cc:1398] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:01.453581 28715 raft_consensus.cc:504] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.453768 28600 tablet_bootstrap.cc:492] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:01.455503 28290 leader_election.cc:341] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: fd98387c1cae45dcb29c2116b6141708
I20240503 14:12:01.455700 28600 tablet_bootstrap.cc:654] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.456076 28715 leader_election.cc:288] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.456916 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d637087c60144aaaa9b0965110d280e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.457594 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d637087c60144aaaa9b0965110d280e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:01.457935 28216 raft_consensus.cc:3037] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.460443 28622 raft_consensus.cc:482] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.460697 28622 raft_consensus.cc:504] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.461289 28216 raft_consensus.cc:2454] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:01.461457 28622 leader_election.cc:288] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.461723 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.462129 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.462822 28653 ts_tablet_manager.cc:1888] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.462998 28653 log.cc:1196] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/fdccb370198b4eb69768aeed787fd682
I20240503 14:12:01.463737 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:01.463824 28290 leader_election.cc:341] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:01.464116 28290 leader_election.cc:341] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: fd98387c1cae45dcb29c2116b6141708
I20240503 14:12:01.464025 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ae10db451b4e3a8a85d1d987c447d9" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.464233 28290 leader_election.cc:302] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.464639 28653 ts_tablet_manager.cc:1909] T fdccb370198b4eb69768aeed787fd682 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.464705 28290 leader_election.cc:302] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.465035 28662 raft_consensus.cc:2726] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:01.465173 28290 leader_election.cc:341] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.465235 28604 tablet_bootstrap.cc:492] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.465291 28662 raft_consensus.cc:2781] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.465417 28662 raft_consensus.cc:686] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:01.465415 28604 ts_tablet_manager.cc:1367] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.008s
I20240503 14:12:01.465723 28662 consensus_queue.cc:237] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.466311 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.466672 28603 tablet_replica.cc:330] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.466454 28604 raft_consensus.cc:348] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.466814 28604 raft_consensus.cc:374] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.466895 28604 raft_consensus.cc:729] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.466890 28603 raft_consensus.cc:2229] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.467083 28603 raft_consensus.cc:2258] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.467231 28604 consensus_queue.cc:260] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.467916 28603 ts_tablet_manager.cc:1875] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.468852 28604 ts_tablet_manager.cc:1398] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:01.469611 28604 tablet_bootstrap.cc:492] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.474531 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 4c6782582fb543f4b621569343995dfc
W20240503 14:12:01.474805 28290 leader_election.cc:341] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 87ae10db451b4e3a8a85d1d987c447d9
I20240503 14:12:01.475147 28290 leader_election.cc:302] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.475260 28604 tablet_bootstrap.cc:654] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.476729 28603 ts_tablet_manager.cc:1888] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.476985 28603 log.cc:1196] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/f9713865e09349ce9f9ae1d4297da2d0
I20240503 14:12:01.477465 28603 ts_tablet_manager.cc:1909] T f9713865e09349ce9f9ae1d4297da2d0 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.479703 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 87ae10db451b4e3a8a85d1d987c447d9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.480626 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.254:36571
--webserver_interface=127.27.50.254
--webserver_port=38435
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 14:12:01.480604 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_870eff51fbc54c9187d45df3f9cbb08f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.481334 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_870eff51fbc54c9187d45df3f9cbb08f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.482416 28600 tablet_bootstrap.cc:492] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.482697 28600 ts_tablet_manager.cc:1367] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240503 14:12:01.482985 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8a55dd8eb2b449d0bf197060444b3571"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.483625 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a55dd8eb2b449d0bf197060444b3571 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.483850 28600 raft_consensus.cc:348] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.484177 28600 raft_consensus.cc:374] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.484336 28600 raft_consensus.cc:729] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.484730 28600 consensus_queue.cc:260] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.486567 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6d7feb39cce84e66b75914a6bcbcfcc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.487277 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d7feb39cce84e66b75914a6bcbcfcc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.488003 28600 ts_tablet_manager.cc:1398] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:01.489001 28600 tablet_bootstrap.cc:492] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.489333 28653 tablet_replica.cc:330] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.489979 28653 raft_consensus.cc:2229] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.490079 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 92c800e997e944158f7c839dd0e00d44
I20240503 14:12:01.490938 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e4c32619dd494186a5a26bbc9cc04dbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.491183 28653 pending_rounds.cc:62] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:01.491526 28653 pending_rounds.cc:68] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597654590210048 op_type: NO_OP noop_request { }
W20240503 14:12:01.491709 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4c32619dd494186a5a26bbc9cc04dbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.493399 28662 raft_consensus.cc:2726] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.496011 28614 raft_consensus.cc:374] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.492070 28653 raft_consensus.cc:2866] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:01.496552 28653 raft_consensus.cc:2258] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.497953 28614 raft_consensus.cc:729] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.498567 28614 consensus_queue.cc:260] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.500461 28614 ts_tablet_manager.cc:1398] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20240503 14:12:01.497814 28600 tablet_bootstrap.cc:654] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.502169 28614 tablet_bootstrap.cc:492] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.505944 28604 tablet_bootstrap.cc:492] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.506157 28604 ts_tablet_manager.cc:1367] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.020s
I20240503 14:12:01.506644 28614 tablet_bootstrap.cc:654] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.507215 28604 raft_consensus.cc:348] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.507460 28604 raft_consensus.cc:374] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.507565 28604 raft_consensus.cc:729] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.507869 28604 consensus_queue.cc:260] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.509604 28604 ts_tablet_manager.cc:1398] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:01.510015 28604 tablet_bootstrap.cc:492] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.512357 28604 tablet_bootstrap.cc:654] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.515290 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c6782582fb543f4b621569343995dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.515377 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c6782582fb543f4b621569343995dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.515635 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ec68f75fa2435698383a03874a5bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.515779 28603 tablet_replica.cc:330] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.516001 28603 raft_consensus.cc:2229] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.516683 28603 pending_rounds.cc:62] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:01.516788 28603 pending_rounds.cc:68] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597655741448192 op_type: NO_OP noop_request { }
I20240503 14:12:01.516927 28603 raft_consensus.cc:2866] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:01.517053 28603 raft_consensus.cc:2258] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.517978 28603 ts_tablet_manager.cc:1875] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.520125 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.520956 28604 tablet_bootstrap.cc:492] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.521189 28604 ts_tablet_manager.cc:1367] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240503 14:12:01.521181 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet f9713865e09349ce9f9ae1d4297da2d0 (table default.Table_36901c8428f346ae8117eb126334063b [id=15661374a8c648338c1386ad0b207148]) successfully deleted
W20240503 14:12:01.521520 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f54dc481079c4a5db5b54d485991fa65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f54dc481079c4a5db5b54d485991fa65 already in progress: creating tablet
W20240503 14:12:01.521811 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f9713865e09349ce9f9ae1d4297da2d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9713865e09349ce9f9ae1d4297da2d0 already in progress: creating tablet
I20240503 14:12:01.522260 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c800e997e944158f7c839dd0e00d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.523350 28604 raft_consensus.cc:348] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.523718 28604 raft_consensus.cc:374] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.523806 28604 raft_consensus.cc:729] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.523926 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.524051 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet f35e87db1c2a42a79b426b2e7767367a with delete_type TABLET_DATA_DELETED (replaced by tablet 7de7832122804eada175f710cc5bab5e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.524365 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet fdccb370198b4eb69768aeed787fd682 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.524434 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet f54dc481079c4a5db5b54d485991fa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.524559 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f15be6b9e4baaba9587c3634f72ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.524681 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c800e997e944158f7c839dd0e00d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.524816 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet fd98387c1cae45dcb29c2116b6141708 with delete_type TABLET_DATA_DELETED (replaced by tablet a2bfdcb8e2b241ca9a6922fd8491e27b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.525091 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet fdccb370198b4eb69768aeed787fd682 (table default.Table_9c60ef813e134942b2bf7a67b5f6ecf4 [id=9f10df997dc04f05a4f052f20c4a2fc5]) successfully deleted
W20240503 14:12:01.525480 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a50237f7cc324e61a27ef6fe2dd7f071 because the tablet was not found. No further retry: Not found: Tablet not found: a50237f7cc324e61a27ef6fe2dd7f071
I20240503 14:12:01.525451 28604 consensus_queue.cc:260] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.526258 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c6782582fb543f4b621569343995dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.526569 28661 tablet_replica.cc:330] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:01.526834 28661 raft_consensus.cc:2229] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.527060 28661 raft_consensus.cc:2258] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.527720 28603 ts_tablet_manager.cc:1888] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:01.527886 28661 ts_tablet_manager.cc:1875] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.527922 28603 log.cc:1196] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/4c6782582fb543f4b621569343995dfc
I20240503 14:12:01.527931 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c800e997e944158f7c839dd0e00d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.528127 28604 ts_tablet_manager.cc:1398] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240503 14:12:01.528523 28603 ts_tablet_manager.cc:1909] T 4c6782582fb543f4b621569343995dfc P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.528981 28604 tablet_bootstrap.cc:492] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.529860 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 4c6782582fb543f4b621569343995dfc (table default.Table_0f8086df11104655afdf191191957937 [id=9f62dbc5baa94df18b25548d731ddeee]) successfully deleted
I20240503 14:12:01.530313 28603 tablet_replica.cc:330] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.530596 28603 raft_consensus.cc:2229] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.531092 28604 tablet_bootstrap.cc:654] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.531247 28603 pending_rounds.cc:62] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:01.531531 28603 pending_rounds.cc:68] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597655005773824 op_type: NO_OP noop_request { }
I20240503 14:12:01.531728 28603 raft_consensus.cc:2866] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:01.534787 28653 ts_tablet_manager.cc:1875] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.535348 28661 ts_tablet_manager.cc:1888] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.535537 28661 log.cc:1196] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/4c6782582fb543f4b621569343995dfc
I20240503 14:12:01.535956 28661 ts_tablet_manager.cc:1909] T 4c6782582fb543f4b621569343995dfc P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:01.536355 28610 raft_consensus.cc:482] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.536547 28610 raft_consensus.cc:504] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.537240 28600 tablet_bootstrap.cc:492] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.537333 28610 leader_election.cc:288] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.538215 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9713865e09349ce9f9ae1d4297da2d0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.538594 28600 ts_tablet_manager.cc:1367] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.007s
I20240503 14:12:01.539662 28600 raft_consensus.cc:348] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:01.539850 28150 leader_election.cc:341] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.540006 28600 raft_consensus.cc:374] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.540143 28600 raft_consensus.cc:729] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.541821 28604 tablet_bootstrap.cc:492] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.542064 28604 ts_tablet_manager.cc:1367] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240503 14:12:01.542588 28653 ts_tablet_manager.cc:1888] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:01.542752 28653 log.cc:1196] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f54dc481079c4a5db5b54d485991fa65
W20240503 14:12:01.542802 28150 leader_election.cc:341] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: f9713865e09349ce9f9ae1d4297da2d0
I20240503 14:12:01.542939 28150 leader_election.cc:302] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.543260 28653 ts_tablet_manager.cc:1909] T f54dc481079c4a5db5b54d485991fa65 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.543458 28610 raft_consensus.cc:2726] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.543397 28604 raft_consensus.cc:348] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.543633 28604 raft_consensus.cc:374] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.543756 28604 raft_consensus.cc:729] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.544034 28604 consensus_queue.cc:260] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.544586 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet f54dc481079c4a5db5b54d485991fa65 (table default.Table_9289210d002542b78ccb48780cf16018 [id=3e4bfd54a342491b86e978c08faa5c8b]) successfully deleted
I20240503 14:12:01.545019 28653 tablet_replica.cc:330] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.545243 28653 raft_consensus.cc:2229] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.545497 28653 pending_rounds.cc:62] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:01.545590 28653 pending_rounds.cc:68] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597655196061696 op_type: NO_OP noop_request { }
I20240503 14:12:01.545734 28653 raft_consensus.cc:2866] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:01.545855 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 4c6782582fb543f4b621569343995dfc (table default.Table_0f8086df11104655afdf191191957937 [id=9f62dbc5baa94df18b25548d731ddeee]) successfully deleted
I20240503 14:12:01.545950 28653 raft_consensus.cc:2258] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.545888 28600 consensus_queue.cc:260] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:01.546248 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 92c800e997e944158f7c839dd0e00d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92c800e997e944158f7c839dd0e00d44 already in progress: creating tablet
I20240503 14:12:01.546674 28604 ts_tablet_manager.cc:1398] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:01.547122 28600 ts_tablet_manager.cc:1398] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 14:12:01.547605 28653 ts_tablet_manager.cc:1875] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.547646 28604 tablet_bootstrap.cc:492] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.548506 28600 tablet_bootstrap.cc:492] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.549882 28604 tablet_bootstrap.cc:654] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.550940 28600 tablet_bootstrap.cc:654] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.531898 28603 raft_consensus.cc:2258] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.553269 28603 ts_tablet_manager.cc:1875] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.553611 28604 tablet_bootstrap.cc:492] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.553614 28614 tablet_bootstrap.cc:492] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.553805 28604 ts_tablet_manager.cc:1367] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240503 14:12:01.553812 28614 ts_tablet_manager.cc:1367] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.001s
I20240503 14:12:01.554702 28604 raft_consensus.cc:348] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.554710 28614 raft_consensus.cc:348] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.554971 28604 raft_consensus.cc:374] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.554970 28614 raft_consensus.cc:374] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.555059 28614 raft_consensus.cc:729] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.555059 28604 raft_consensus.cc:729] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.555356 28614 consensus_queue.cc:260] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.555500 28604 consensus_queue.cc:260] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.556114 28604 ts_tablet_manager.cc:1398] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:01.556488 28604 tablet_bootstrap.cc:492] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.558773 28604 tablet_bootstrap.cc:654] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.559527 28614 ts_tablet_manager.cc:1398] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:01.561290 28614 tablet_bootstrap.cc:492] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.564972 28653 ts_tablet_manager.cc:1888] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:01.565181 28653 log.cc:1196] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6aa994dfa54a47408bfb34bcfda9a415
I20240503 14:12:01.565753 28653 ts_tablet_manager.cc:1909] T 6aa994dfa54a47408bfb34bcfda9a415 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.567108 28653 tablet_replica.cc:330] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.567059 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6aa994dfa54a47408bfb34bcfda9a415 (table default.Table_4e83fe8e46644b8f8df871f6da4f0f1d [id=a1a30b8a5adb44b390e6bea2dd82c20c]) successfully deleted
I20240503 14:12:01.567350 28653 raft_consensus.cc:2229] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.567545 28653 raft_consensus.cc:2258] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.571133 28653 ts_tablet_manager.cc:1875] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.575027 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fbe6764ce0f84db08b3d9ef65525d733" } new_table_name: "default.Table_f2e8bc9e82a64d6682fa813e04fe7fc4" modify_external_catalogs: true
I20240503 14:12:01.576800 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2b1373375e074bbba74f7bb80cf43abf" } new_table_name: "default.Table_73db66f21afd468d887601ce8148489a" modify_external_catalogs: true
I20240503 14:12:01.578428 28031 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet cf95d9731dbb488e908831b4c9d4f1d9
I20240503 14:12:01.578794 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet f9713865e09349ce9f9ae1d4297da2d0 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.579507 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_30f21f3fef874fdf94af3d3d043de7d7" } modify_external_catalogs: true
I20240503 14:12:01.579752 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_30f21f3fef874fdf94af3d3d043de7d7" } modify_external_catalogs: true
I20240503 14:12:01.580641 28614 tablet_bootstrap.cc:654] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.580811 28653 ts_tablet_manager.cc:1888] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.581048 28653 log.cc:1196] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f9713865e09349ce9f9ae1d4297da2d0
I20240503 14:12:01.581450 28653 ts_tablet_manager.cc:1909] T f9713865e09349ce9f9ae1d4297da2d0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.582598 28600 tablet_bootstrap.cc:492] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.582758 28653 tablet_replica.cc:330] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.582777 28600 ts_tablet_manager.cc:1367] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.003s
I20240503 14:12:01.582998 28653 raft_consensus.cc:2229] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.583204 28653 raft_consensus.cc:2258] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.583616 28604 tablet_bootstrap.cc:492] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.583794 28604 ts_tablet_manager.cc:1367] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.016s
I20240503 14:12:01.583817 28653 ts_tablet_manager.cc:1875] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.583742 28600 raft_consensus.cc:348] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.583997 28600 raft_consensus.cc:374] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.584128 28600 raft_consensus.cc:729] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.584394 28600 consensus_queue.cc:260] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.585000 28604 raft_consensus.cc:348] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.585237 28604 raft_consensus.cc:374] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.585332 28604 raft_consensus.cc:729] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.585595 28604 consensus_queue.cc:260] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.586184 28604 ts_tablet_manager.cc:1398] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:01.586931 28604 tablet_bootstrap.cc:492] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.587474 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet f9713865e09349ce9f9ae1d4297da2d0 (table default.Table_36901c8428f346ae8117eb126334063b [id=15661374a8c648338c1386ad0b207148]) successfully deleted
I20240503 14:12:01.589836 28600 ts_tablet_manager.cc:1398] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:01.592171 28600 tablet_bootstrap.cc:492] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.594501 28600 tablet_bootstrap.cc:654] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.596772 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8d906bfe77124e609d331427b67fabd0" } new_table_name: "default.Table_22bc40c64b7d4c8f8040ce17b7580930" modify_external_catalogs: true
I20240503 14:12:01.598698 28604 tablet_bootstrap.cc:654] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.599960 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet f35e87db1c2a42a79b426b2e7767367a from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.600107 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_55fe0191399e49d19d8ffd2e6cd0a77c" } new_table_name: "default.Table_41714410a87e4a7cacd1dd22125c433f" modify_external_catalogs: true
I20240503 14:12:01.600613 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 2502a13e8cbc4a3abfed225f94c44d2f from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.601234 28600 tablet_bootstrap.cc:492] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.601480 28600 ts_tablet_manager.cc:1367] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:12:01.603031 28614 tablet_bootstrap.cc:492] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.603216 28614 ts_tablet_manager.cc:1367] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.005s
I20240503 14:12:01.604115 28614 raft_consensus.cc:348] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.604305 28634 raft_consensus.cc:482] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.604374 28614 raft_consensus.cc:374] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.604485 28614 raft_consensus.cc:729] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.604481 28634 raft_consensus.cc:504] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.604753 28614 consensus_queue.cc:260] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.605240 28634 leader_election.cc:288] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.605540 28653 ts_tablet_manager.cc:1888] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.605697 28653 log.cc:1196] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/4c6782582fb543f4b621569343995dfc
I20240503 14:12:01.606129 28653 ts_tablet_manager.cc:1909] T 4c6782582fb543f4b621569343995dfc P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.606158 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d637087c60144aaaa9b0965110d280e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:01.606478 28216 raft_consensus.cc:2379] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:01.607522 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d637087c60144aaaa9b0965110d280e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:01.608588 28427 leader_election.cc:302] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:01.609290 28634 raft_consensus.cc:3037] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.609607 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 4c6782582fb543f4b621569343995dfc (table default.Table_0f8086df11104655afdf191191957937 [id=9f62dbc5baa94df18b25548d731ddeee]) successfully deleted
I20240503 14:12:01.609789 28614 ts_tablet_manager.cc:1398] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
W20240503 14:12:01.610023 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 92c800e997e944158f7c839dd0e00d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92c800e997e944158f7c839dd0e00d44 already in progress: creating tablet
I20240503 14:12:01.611865 28614 tablet_bootstrap.cc:492] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:01.612833 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fe5d280a1654d80a2176c1def459e2e consensus state has no leader
I20240503 14:12:01.613798 28614 tablet_bootstrap.cc:654] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.615298 28600 raft_consensus.cc:348] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.615559 28600 raft_consensus.cc:374] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.615638 28600 raft_consensus.cc:729] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.616335 28634 raft_consensus.cc:2726] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.615913 28600 consensus_queue.cc:260] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.617736 28604 tablet_bootstrap.cc:492] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.618168 28604 ts_tablet_manager.cc:1367] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.001s
I20240503 14:12:01.618245 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d56c053e47bb4621b5a7d74ca45069cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.618395 28622 raft_consensus.cc:482] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.618585 28622 raft_consensus.cc:504] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:01.618975 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d56c053e47bb4621b5a7d74ca45069cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.619360 28622 leader_election.cc:288] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.620255 28604 raft_consensus.cc:348] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.620533 28604 raft_consensus.cc:374] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.620548 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f15be6b9e4baaba9587c3634f72ad" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:01.620628 28604 raft_consensus.cc:729] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.620592 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f15be6b9e4baaba9587c3634f72ad" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.620800 28216 raft_consensus.cc:2454] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:01.620834 28494 raft_consensus.cc:2454] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:01.620921 28604 consensus_queue.cc:260] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.621354 28290 leader_election.cc:302] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:01.621577 28604 ts_tablet_manager.cc:1398] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:01.621968 28604 tablet_bootstrap.cc:492] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.622443 28622 raft_consensus.cc:2781] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.622577 28622 raft_consensus.cc:482] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.622675 28622 raft_consensus.cc:3037] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.624682 28604 tablet_bootstrap.cc:654] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.626597 28622 raft_consensus.cc:504] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.627566 28622 leader_election.cc:288] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.628618 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f15be6b9e4baaba9587c3634f72ad" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.628664 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f15be6b9e4baaba9587c3634f72ad" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:01.628890 28494 raft_consensus.cc:3037] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.628942 28216 raft_consensus.cc:3037] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.632395 28494 raft_consensus.cc:2454] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:01.632443 28216 raft_consensus.cc:2454] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:01.633131 28290 leader_election.cc:302] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:01.634043 28622 raft_consensus.cc:2781] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.635500 28622 raft_consensus.cc:686] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
W20240503 14:12:01.636374 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc337647c7cd402e91720478380f4696 has no consensus state
I20240503 14:12:01.635879 28622 consensus_queue.cc:237] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.637938 28600 ts_tablet_manager.cc:1398] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 14:12:01.639794 28634 raft_consensus.cc:482] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.641937 28600 tablet_bootstrap.cc:492] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.646057 28600 tablet_bootstrap.cc:654] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.651692 28634 raft_consensus.cc:504] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.652686 28634 leader_election.cc:288] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.655834 28600 tablet_bootstrap.cc:492] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
W20240503 14:12:01.655928 28427 leader_election.cc:341] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: f54dc481079c4a5db5b54d485991fa65
I20240503 14:12:01.656045 28600 ts_tablet_manager.cc:1367] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
W20240503 14:12:01.657068 28427 leader_election.cc:341] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: f54dc481079c4a5db5b54d485991fa65
I20240503 14:12:01.657039 28600 raft_consensus.cc:348] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.657239 28427 leader_election.cc:302] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:01.657294 28600 raft_consensus.cc:374] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.657382 28600 raft_consensus.cc:729] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.658175 28634 raft_consensus.cc:2726] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.658514 28600 consensus_queue.cc:260] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.661806 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_adb8e4f58ba042f0a57523a25228ca92" } modify_external_catalogs: true
I20240503 14:12:01.662156 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_adb8e4f58ba042f0a57523a25228ca92" } modify_external_catalogs: true
I20240503 14:12:01.664700 28600 ts_tablet_manager.cc:1398] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:01.666785 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2502a13e8cbc4a3abfed225f94c44d2f
I20240503 14:12:01.667354 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 2502a13e8cbc4a3abfed225f94c44d2f deleted and replaced by tablet da19787034c24ebe929601d49cb21b3b
I20240503 14:12:01.669580 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f with delete_type TABLET_DATA_DELETED (replaced by tablet da19787034c24ebe929601d49cb21b3b) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.669844 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f with delete_type TABLET_DATA_DELETED (replaced by tablet da19787034c24ebe929601d49cb21b3b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.670172 28653 tablet_replica.cc:330] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.670404 28653 raft_consensus.cc:2229] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.670449 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f with delete_type TABLET_DATA_DELETED (replaced by tablet da19787034c24ebe929601d49cb21b3b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.670617 28653 raft_consensus.cc:2258] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.671731 28653 ts_tablet_manager.cc:1875] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.672001 28622 raft_consensus.cc:482] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.672242 28622 raft_consensus.cc:504] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.672518 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2ec7eed485554ccbb4bd34b53bdc7618" } new_table_name: "default.Table_cc4fe0dac143484396ef862744cc9967" modify_external_catalogs: true
I20240503 14:12:01.673358 28622 leader_election.cc:288] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:01.673542 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50237f7cc324e61a27ef6fe2dd7f071" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:01.673759 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2502a13e8cbc4a3abfed225f94c44d2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2502a13e8cbc4a3abfed225f94c44d2f already in progress: creating tablet
I20240503 14:12:01.673820 28494 raft_consensus.cc:2454] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:01.674293 28290 leader_election.cc:302] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:01.674625 28622 raft_consensus.cc:2781] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.674708 28622 raft_consensus.cc:482] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.674804 28622 raft_consensus.cc:3037] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.676223 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
W20240503 14:12:01.677098 28290 leader_election.cc:341] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a50237f7cc324e61a27ef6fe2dd7f071
I20240503 14:12:01.678133 28622 raft_consensus.cc:504] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.677906 28026 catalog_manager.cc:5484] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:01.678476 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 5d637087c60144aaaa9b0965110d280e (table default.Table_55fe0191399e49d19d8ffd2e6cd0a77c [id=500a3febdac042138ea31f6f3c8d02b5]). Expected version 2 got 0
I20240503 14:12:01.678857 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
I20240503 14:12:01.679231 28024 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b] alter complete (version 3)
I20240503 14:12:01.679270 28622 leader_election.cc:288] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:01.680068 28290 leader_election.cc:341] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a50237f7cc324e61a27ef6fe2dd7f071
I20240503 14:12:01.680172 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50237f7cc324e61a27ef6fe2dd7f071" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:01.680466 28494 raft_consensus.cc:3037] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.680495 28653 ts_tablet_manager.cc:1888] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.680655 28653 log.cc:1196] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2502a13e8cbc4a3abfed225f94c44d2f
I20240503 14:12:01.680809 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.681068 28653 ts_tablet_manager.cc:1909] T 2502a13e8cbc4a3abfed225f94c44d2f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.681157 28661 tablet_replica.cc:330] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:01.681365 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d637087c60144aaaa9b0965110d280e consensus state has no leader
I20240503 14:12:01.681372 28661 raft_consensus.cc:2229] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.681596 28661 raft_consensus.cc:2258] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.682236 28661 ts_tablet_manager.cc:1875] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.682384 28600 tablet_bootstrap.cc:492] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.684962 28494 raft_consensus.cc:2454] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:01.684059 28600 tablet_bootstrap.cc:654] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.687811 28290 leader_election.cc:302] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:01.689131 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c0611cf7aa4ff3b4e7e6081558729b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.691545 28603 ts_tablet_manager.cc:1888] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:01.691793 28603 log.cc:1196] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/67ec68f75fa2435698383a03874a5bae
I20240503 14:12:01.691884 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ec68f75fa2435698383a03874a5bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.692341 28603 ts_tablet_manager.cc:1909] T 67ec68f75fa2435698383a03874a5bae P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.692793 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.693799 28471 tablet_service.cc:1467] Processing CreateTablet for tablet ec6186c8f2e546f39bf0f632d76166fa (DEFAULT_TABLE table=default.Table_3c2d3cac1d8c4aa4b9730b7f0baa45a0 [id=b1c63696396c4737b11ac03b53d4160e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.694307 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6186c8f2e546f39bf0f632d76166fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.696792 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7afc535ef1bb440493a9e65dc3c86fde" } modify_external_catalogs: true
I20240503 14:12:01.697083 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7afc535ef1bb440493a9e65dc3c86fde" } modify_external_catalogs: true
I20240503 14:12:01.699223 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8543af24474940c3b1ed368e33cba893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.699841 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8543af24474940c3b1ed368e33cba893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.700763 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7d8e4d1ecd794871b9d73289f8f43c9a" } modify_external_catalogs: true
I20240503 14:12:01.700970 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7d8e4d1ecd794871b9d73289f8f43c9a" } modify_external_catalogs: true
I20240503 14:12:01.701529 28692 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:01.703974 28330 tablet_service.cc:1467] Processing CreateTablet for tablet ec6186c8f2e546f39bf0f632d76166fa (DEFAULT_TABLE table=default.Table_3c2d3cac1d8c4aa4b9730b7f0baa45a0 [id=b1c63696396c4737b11ac03b53d4160e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.703960 28192 tablet_service.cc:1467] Processing CreateTablet for tablet ec6186c8f2e546f39bf0f632d76166fa (DEFAULT_TABLE table=default.Table_3c2d3cac1d8c4aa4b9730b7f0baa45a0 [id=b1c63696396c4737b11ac03b53d4160e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.704233 28711 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:01.704294 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_59af1c61c0534502b0f2cdaff98f4e96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.704465 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6186c8f2e546f39bf0f632d76166fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.704473 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6186c8f2e546f39bf0f632d76166fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:01.704907 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59af1c61c0534502b0f2cdaff98f4e96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.710515 28603 tablet_replica.cc:330] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.712201 28603 raft_consensus.cc:2229] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.712935 28603 pending_rounds.cc:62] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:01.714501 28603 pending_rounds.cc:68] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597654579236864 op_type: NO_OP noop_request { }
I20240503 14:12:01.715130 28603 raft_consensus.cc:2866] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:01.715696 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (DEFAULT_TABLE table=default.Table_9d94370dfab6449ea9ad79c5530f078a [id=8961b0b7af2a4e999ee705db8d8e2bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.716022 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (DEFAULT_TABLE table=default.Table_9d94370dfab6449ea9ad79c5530f078a [id=8961b0b7af2a4e999ee705db8d8e2bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.716221 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c875ec8c5e49d1bb20a23dd7a21902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.716504 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c875ec8c5e49d1bb20a23dd7a21902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.716629 28603 raft_consensus.cc:2258] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.717262 28336 tablet_service.cc:1467] Processing CreateTablet for tablet bacf2d1091ee458c961c691093ea65a9 (DEFAULT_TABLE table=default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.717571 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2502a13e8cbc4a3abfed225f94c44d2f (table default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:01.717682 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bacf2d1091ee458c961c691093ea65a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.718329 28335 tablet_service.cc:1467] Processing CreateTablet for tablet ab05178dc081480ba409aab7f143f849 (DEFAULT_TABLE table=default.Table_adb8e4f58ba042f0a57523a25228ca92 [id=37c31c087c65469f96f4322d84ba161f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.718694 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab05178dc081480ba409aab7f143f849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.715250 28705 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:01.720432 28467 tablet_service.cc:1467] Processing CreateTablet for tablet bacf2d1091ee458c961c691093ea65a9 (DEFAULT_TABLE table=default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.720885 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bacf2d1091ee458c961c691093ea65a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.724782 28473 tablet_service.cc:1467] Processing CreateTablet for tablet ab05178dc081480ba409aab7f143f849 (DEFAULT_TABLE table=default.Table_adb8e4f58ba042f0a57523a25228ca92 [id=37c31c087c65469f96f4322d84ba161f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.724866 28197 tablet_service.cc:1467] Processing CreateTablet for tablet ab05178dc081480ba409aab7f143f849 (DEFAULT_TABLE table=default.Table_adb8e4f58ba042f0a57523a25228ca92 [id=37c31c087c65469f96f4322d84ba161f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.725219 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab05178dc081480ba409aab7f143f849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.725289 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab05178dc081480ba409aab7f143f849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.727816 28603 ts_tablet_manager.cc:1875] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.728598 28189 tablet_service.cc:1467] Processing CreateTablet for tablet bacf2d1091ee458c961c691093ea65a9 (DEFAULT_TABLE table=default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.729666 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bacf2d1091ee458c961c691093ea65a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.735172 28634 raft_consensus.cc:482] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.735515 28634 raft_consensus.cc:504] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.736268 28634 leader_election.cc:288] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:01.737311 28427 leader_election.cc:341] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 87ae10db451b4e3a8a85d1d987c447d9
I20240503 14:12:01.739835 28661 ts_tablet_manager.cc:1888] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.740307 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ae10db451b4e3a8a85d1d987c447d9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:01.740629 28357 raft_consensus.cc:2454] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:01.741282 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 7de7832122804eada175f710cc5bab5e (DEFAULT_TABLE table=default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.741876 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de7832122804eada175f710cc5bab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.742555 28603 ts_tablet_manager.cc:1888] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:01.742785 28603 log.cc:1196] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:01.743371 28603 ts_tablet_manager.cc:1909] T 71598d0a44384a3b93dec30d6c60ecd5 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.716004 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (DEFAULT_TABLE table=default.Table_9d94370dfab6449ea9ad79c5530f078a [id=8961b0b7af2a4e999ee705db8d8e2bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.744799 28603 tablet_replica.cc:330] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.745131 28603 raft_consensus.cc:2229] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.745424 28603 raft_consensus.cc:2258] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.746076 28603 ts_tablet_manager.cc:1875] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.746104 28600 tablet_bootstrap.cc:492] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.745105 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c875ec8c5e49d1bb20a23dd7a21902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.746286 28600 ts_tablet_manager.cc:1367] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.005s
I20240503 14:12:01.747210 28600 raft_consensus.cc:348] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.747484 28600 raft_consensus.cc:374] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.747658 28600 raft_consensus.cc:729] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.741127 28427 leader_election.cc:302] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:01.748046 28600 consensus_queue.cc:260] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.748435 28634 raft_consensus.cc:2781] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.748564 28634 raft_consensus.cc:482] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.748682 28634 raft_consensus.cc:3037] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.749403 28614 tablet_bootstrap.cc:492] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:01.749591 28614 ts_tablet_manager.cc:1367] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.005s
I20240503 14:12:01.750622 28614 raft_consensus.cc:348] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.750931 28614 raft_consensus.cc:374] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.751019 28614 raft_consensus.cc:729] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.751353 28634 raft_consensus.cc:504] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.751343 28614 consensus_queue.cc:260] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.751744 28715 raft_consensus.cc:2781] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.751914 28715 raft_consensus.cc:686] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:01.752435 28634 leader_election.cc:288] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.752960 28715 consensus_queue.cc:237] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.753944 28661 log.cc:1196] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:01.754652 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ae10db451b4e3a8a85d1d987c447d9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:01.754894 28357 raft_consensus.cc:3037] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.755764 28661 ts_tablet_manager.cc:1909] T 71598d0a44384a3b93dec30d6c60ecd5 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:01.755720 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 7de7832122804eada175f710cc5bab5e (DEFAULT_TABLE table=default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.756212 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de7832122804eada175f710cc5bab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.757645 28604 tablet_bootstrap.cc:492] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.757822 28604 ts_tablet_manager.cc:1367] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.136s user 0.022s sys 0.017s
I20240503 14:12:01.757946 28357 raft_consensus.cc:2454] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:01.758545 28427 leader_election.cc:302] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:01.758872 28634 raft_consensus.cc:2781] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
W20240503 14:12:01.759245 28427 leader_election.cc:341] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 87ae10db451b4e3a8a85d1d987c447d9
I20240503 14:12:01.759279 28603 ts_tablet_manager.cc:1888] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.759424 28603 log.cc:1196] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/38c0611cf7aa4ff3b4e7e6081558729b
I20240503 14:12:01.761883 28603 ts_tablet_manager.cc:1909] T 38c0611cf7aa4ff3b4e7e6081558729b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.762542 28467 tablet_service.cc:1467] Processing CreateTablet for tablet b1de9717d0e44179b875174d5a76e4c3 (DEFAULT_TABLE table=default.Table_5a5d61e70c514fbfac570c0d594e7010 [id=e2f2ebf7731a41e3ae451eddf555c454]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.763031 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1de9717d0e44179b875174d5a76e4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.763036 28330 tablet_service.cc:1467] Processing CreateTablet for tablet b1de9717d0e44179b875174d5a76e4c3 (DEFAULT_TABLE table=default.Table_5a5d61e70c514fbfac570c0d594e7010 [id=e2f2ebf7731a41e3ae451eddf555c454]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.763006 28192 tablet_service.cc:1467] Processing CreateTablet for tablet b1de9717d0e44179b875174d5a76e4c3 (DEFAULT_TABLE table=default.Table_5a5d61e70c514fbfac570c0d594e7010 [id=e2f2ebf7731a41e3ae451eddf555c454]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.763407 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1de9717d0e44179b875174d5a76e4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.763479 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1de9717d0e44179b875174d5a76e4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.763672 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef6cd66f34c4980bad5e500309b1568 (DEFAULT_TABLE table=default.Table_f3298a69afb440bc9a9c865eb05e070d [id=9e37697ebba24976a5f031ba8e9d87e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.764122 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef6cd66f34c4980bad5e500309b1568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.766377 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef6cd66f34c4980bad5e500309b1568 (DEFAULT_TABLE table=default.Table_f3298a69afb440bc9a9c865eb05e070d [id=9e37697ebba24976a5f031ba8e9d87e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.766402 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef6cd66f34c4980bad5e500309b1568 (DEFAULT_TABLE table=default.Table_f3298a69afb440bc9a9c865eb05e070d [id=9e37697ebba24976a5f031ba8e9d87e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.766806 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef6cd66f34c4980bad5e500309b1568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.766805 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef6cd66f34c4980bad5e500309b1568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.767040 28473 tablet_service.cc:1467] Processing CreateTablet for tablet dc337647c7cd402e91720478380f4696 (DEFAULT_TABLE table=default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.767446 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc337647c7cd402e91720478380f4696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.767596 28189 tablet_service.cc:1467] Processing CreateTablet for tablet dc337647c7cd402e91720478380f4696 (DEFAULT_TABLE table=default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.767979 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc337647c7cd402e91720478380f4696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.768680 28604 raft_consensus.cc:348] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.768908 28604 raft_consensus.cc:374] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.769006 28604 raft_consensus.cc:729] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.769250 28604 consensus_queue.cc:260] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.769855 28661 tablet_replica.cc:330] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:01.770977 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 23186a7a0dfb432e813c8a05e97e46dd (DEFAULT_TABLE table=default.Table_976081c8371b4dc68cb226625ef1b6c6 [id=68a4a475afc24b95b5388e43deb4632b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.771453 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23186a7a0dfb432e813c8a05e97e46dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.772533 28331 tablet_service.cc:1467] Processing CreateTablet for tablet dc337647c7cd402e91720478380f4696 (DEFAULT_TABLE table=default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.772944 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc337647c7cd402e91720478380f4696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.773499 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 23186a7a0dfb432e813c8a05e97e46dd (DEFAULT_TABLE table=default.Table_976081c8371b4dc68cb226625ef1b6c6 [id=68a4a475afc24b95b5388e43deb4632b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.773933 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23186a7a0dfb432e813c8a05e97e46dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.776502 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 23186a7a0dfb432e813c8a05e97e46dd (DEFAULT_TABLE table=default.Table_976081c8371b4dc68cb226625ef1b6c6 [id=68a4a475afc24b95b5388e43deb4632b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.782599 28603 tablet_replica.cc:330] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.782862 28603 raft_consensus.cc:2229] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.783102 28603 raft_consensus.cc:2258] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.784693 28614 ts_tablet_manager.cc:1398] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20240503 14:12:01.786062 28614 tablet_bootstrap.cc:492] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:01.787487 28603 ts_tablet_manager.cc:1875] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.789804 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23186a7a0dfb432e813c8a05e97e46dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.790096 28468 tablet_service.cc:1467] Processing CreateTablet for tablet f45eb624434a404d85dcd233721ebf68 (DEFAULT_TABLE table=default.Table_45bc934b7dba4fbc966160c359918f67 [id=a61553d45d10436eb38fd35c53417842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.791185 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f45eb624434a404d85dcd233721ebf68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.762079 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 7de7832122804eada175f710cc5bab5e (DEFAULT_TABLE table=default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.792608 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de7832122804eada175f710cc5bab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.796782 28603 ts_tablet_manager.cc:1888] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.797035 28603 log.cc:1196] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/f35e87db1c2a42a79b426b2e7767367a
I20240503 14:12:01.797520 28603 ts_tablet_manager.cc:1909] T f35e87db1c2a42a79b426b2e7767367a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.798542 28622 raft_consensus.cc:482] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.798786 28603 tablet_replica.cc:330] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.798835 28622 raft_consensus.cc:504] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.799126 28603 raft_consensus.cc:2229] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.799458 28603 pending_rounds.cc:62] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:01.799594 28603 pending_rounds.cc:68] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597656625221632 op_type: NO_OP noop_request { }
I20240503 14:12:01.799782 28603 raft_consensus.cc:2866] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:01.799983 28603 raft_consensus.cc:2258] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.800119 28622 leader_election.cc:288] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:01.800590 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85b6db1a21c4bca80c8bdec17e45c59" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.800882 28661 raft_consensus.cc:2229] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.800882 28603 ts_tablet_manager.cc:1875] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.801090 28661 raft_consensus.cc:2258] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.802721 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 67ec68f75fa2435698383a03874a5bae (table default.Table_a9c4dd16181e4b0789779c3f25eaed1c [id=18925bb0895246988de55da2f108454b]) successfully deleted
I20240503 14:12:01.803390 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 71598d0a44384a3b93dec30d6c60ecd5 (table default.Table_2be03633b8854e03955f07503a5309a5 [id=7907cfab39b74beba4ceddd1762bffcd]) successfully deleted
I20240503 14:12:01.803892 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 38c0611cf7aa4ff3b4e7e6081558729b (table default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e [id=e6b422e744c14453a7e7c4b28c1eaac1]) successfully deleted
I20240503 14:12:01.803912 28190 tablet_service.cc:1467] Processing CreateTablet for tablet f45eb624434a404d85dcd233721ebf68 (DEFAULT_TABLE table=default.Table_45bc934b7dba4fbc966160c359918f67 [id=a61553d45d10436eb38fd35c53417842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.804451 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f45eb624434a404d85dcd233721ebf68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.804895 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85b6db1a21c4bca80c8bdec17e45c59" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:01.805152 28216 raft_consensus.cc:2454] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:01.806551 28330 tablet_service.cc:1467] Processing CreateTablet for tablet f45eb624434a404d85dcd233721ebf68 (DEFAULT_TABLE table=default.Table_45bc934b7dba4fbc966160c359918f67 [id=a61553d45d10436eb38fd35c53417842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:01.806797 28634 raft_consensus.cc:686] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:01.807224 28634 consensus_queue.cc:237] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.809115 28604 ts_tablet_manager.cc:1398] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.051s user 0.003s sys 0.000s
I20240503 14:12:01.810048 28604 tablet_bootstrap.cc:492] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.812247 28604 tablet_bootstrap.cc:654] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.817251 28600 ts_tablet_manager.cc:1398] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.071s user 0.003s sys 0.000s
I20240503 14:12:01.821089 28610 raft_consensus.cc:482] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.821321 28610 raft_consensus.cc:504] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.822178 28610 leader_election.cc:288] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.824414 28600 tablet_bootstrap.cc:492] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.824669 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f45eb624434a404d85dcd233721ebf68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.804347 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet f35e87db1c2a42a79b426b2e7767367a (table default.Table_2ec7eed485554ccbb4bd34b53bdc7618 [id=e324426e1ff447a38c25b3867724fc4c]) successfully deleted
W20240503 14:12:01.827791 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7fe5d280a1654d80a2176c1def459e2e (table default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7fe5d280a1654d80a2176c1def459e2e consensus state has no leader
W20240503 14:12:01.828960 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet fdccb370198b4eb69768aeed787fd682 because the tablet was not found. No further retry: Not found: Tablet not found: fdccb370198b4eb69768aeed787fd682
W20240503 14:12:01.835695 28290 leader_election.cc:341] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.837883 28290 leader_election.cc:302] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.838505 28622 raft_consensus.cc:2781] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:12:01.838462 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f54dc481079c4a5db5b54d485991fa65 because the tablet was not found. No further retry: Not found: Tablet not found: f54dc481079c4a5db5b54d485991fa65
I20240503 14:12:01.838434 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb79edba14b646ef9be239b05a291d71" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:01.839438 28150 leader_election.cc:341] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.841131 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 71598d0a44384a3b93dec30d6c60ecd5 (table default.Table_2be03633b8854e03955f07503a5309a5 [id=7907cfab39b74beba4ceddd1762bffcd]) successfully deleted
I20240503 14:12:01.845042 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ab05178dc081480ba409aab7f143f849
I20240503 14:12:01.845779 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 38c0611cf7aa4ff3b4e7e6081558729b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.846293 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet ec6186c8f2e546f39bf0f632d76166fa from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:01.846737 28600 tablet_bootstrap.cc:654] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.812649 28661 ts_tablet_manager.cc:1875] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.848620 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet ab05178dc081480ba409aab7f143f849 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.848693 28604 tablet_bootstrap.cc:492] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.848960 28604 ts_tablet_manager.cc:1367] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.004s
I20240503 14:12:01.849629 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet ab05178dc081480ba409aab7f143f849 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:01.850541 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ab05178dc081480ba409aab7f143f849 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab05178dc081480ba409aab7f143f849 already in progress: creating tablet
I20240503 14:12:01.850891 28604 raft_consensus.cc:348] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.851226 28604 raft_consensus.cc:374] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.851302 28604 raft_consensus.cc:729] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.851593 28604 consensus_queue.cc:260] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.852205 28604 ts_tablet_manager.cc:1398] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:01.852541 28604 tablet_bootstrap.cc:492] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.823954 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb79edba14b646ef9be239b05a291d71" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.856168 28604 tablet_bootstrap.cc:654] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.858184 28600 tablet_bootstrap.cc:492] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:01.858387 28600 ts_tablet_manager.cc:1367] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240503 14:12:01.859303 28600 raft_consensus.cc:348] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.859573 28600 raft_consensus.cc:374] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.859655 28600 raft_consensus.cc:729] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.859906 28600 consensus_queue.cc:260] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:01.860414 28150 leader_election.cc:341] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.860615 28150 leader_election.cc:302] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.861482 28610 raft_consensus.cc:2726] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.866585 28604 tablet_bootstrap.cc:492] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.866777 28604 ts_tablet_manager.cc:1367] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240503 14:12:01.867645 28604 raft_consensus.cc:348] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.867861 28604 raft_consensus.cc:374] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.867944 28604 raft_consensus.cc:729] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.872895 28027 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 5d637087c60144aaaa9b0965110d280e (table default.Table_41714410a87e4a7cacd1dd22125c433f [id=500a3febdac042138ea31f6f3c8d02b5]). Expected version 3 got 0
I20240503 14:12:01.873044 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 559342d357d64f9eb9e3ab42ccb75942
I20240503 14:12:01.872876 28604 consensus_queue.cc:260] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.873629 28604 ts_tablet_manager.cc:1398] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:01.874064 28604 tablet_bootstrap.cc:492] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.874303 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2fbf7b7a2fb34274a2a21b61197964d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.874545 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7de7832122804eada175f710cc5bab5e consensus state has no leader
W20240503 14:12:01.874944 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fbf7b7a2fb34274a2a21b61197964d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.875015 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7b0e6de295714677a8c47970ae8d75e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.875540 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b0e6de295714677a8c47970ae8d75e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.875865 28604 tablet_bootstrap.cc:654] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.876346 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e85b6db1a21c4bca80c8bdec17e45c59
I20240503 14:12:01.877256 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b47d551510f1443382dad930d5505634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.877869 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b47d551510f1443382dad930d5505634 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.879617 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4033322e04a3488cbc8be0fe6a61bec1" } modify_external_catalogs: true
I20240503 14:12:01.879896 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4033322e04a3488cbc8be0fe6a61bec1" } modify_external_catalogs: true
I20240503 14:12:01.879815 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f99793320ab94248abb70c15ee57ca02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.882220 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_da6c5739b62c43758e9e53182b790622"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.882838 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f99793320ab94248abb70c15ee57ca02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:01.882902 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da6c5739b62c43758e9e53182b790622 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.887924 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet ab05178dc081480ba409aab7f143f849 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.888903 28604 tablet_bootstrap.cc:492] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:01.889133 28604 ts_tablet_manager.cc:1367] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240503 14:12:01.890771 28603 ts_tablet_manager.cc:1888] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:01.891065 28603 log.cc:1196] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/567f15be6b9e4baaba9587c3634f72ad
I20240503 14:12:01.891994 28604 raft_consensus.cc:348] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.892359 28604 raft_consensus.cc:374] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.892944 28622 raft_consensus.cc:482] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.893200 28622 raft_consensus.cc:3037] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.892457 28604 raft_consensus.cc:729] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.893606 28614 tablet_bootstrap.cc:654] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.893697 28604 consensus_queue.cc:260] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:01.894335 28604 ts_tablet_manager.cc:1398] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:01.896307 28604 tablet_bootstrap.cc:492] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:01.898357 28661 ts_tablet_manager.cc:1888] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.898849 28603 ts_tablet_manager.cc:1909] T 567f15be6b9e4baaba9587c3634f72ad P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.899987 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.900918 28603 tablet_replica.cc:330] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.901960 28661 log.cc:1196] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/38c0611cf7aa4ff3b4e7e6081558729b
I20240503 14:12:01.902843 28603 raft_consensus.cc:2229] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.899152 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.903056 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.903822 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet f54dc481079c4a5db5b54d485991fa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.903853 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.903999 28603 raft_consensus.cc:2258] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.904218 28653 tablet_replica.cc:330] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.904423 28661 ts_tablet_manager.cc:1909] T 38c0611cf7aa4ff3b4e7e6081558729b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:01.904430 28653 raft_consensus.cc:2229] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.904636 28653 raft_consensus.cc:2258] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.904958 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 567f15be6b9e4baaba9587c3634f72ad (table default.Table_666d072a7d2a4affa01a3cf64abf777c [id=c094b16f94d44132bea89dc256965971]) successfully deleted
W20240503 14:12:01.905259 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 92c800e997e944158f7c839dd0e00d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92c800e997e944158f7c839dd0e00d44 already in progress: creating tablet
I20240503 14:12:01.905297 28653 ts_tablet_manager.cc:1875] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.908828 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.908835 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet f35e87db1c2a42a79b426b2e7767367a with delete_type TABLET_DATA_DELETED (replaced by tablet 7de7832122804eada175f710cc5bab5e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.909452 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 with delete_type TABLET_DATA_DELETED (replaced by tablet bacf2d1091ee458c961c691093ea65a9) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.909749 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f15be6b9e4baaba9587c3634f72ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.910060 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.912601 27998 master_runner.cc:186] Error getting master registration for 127.27.50.254:36571: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111),
I20240503 14:12:01.912778 27998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 14:12:01.913710 28600 ts_tablet_manager.cc:1398] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.055s user 0.003s sys 0.000s
I20240503 14:12:01.903749 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.901613 28622 raft_consensus.cc:504] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.915134 28604 tablet_bootstrap.cc:654] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.917322 28603 ts_tablet_manager.cc:1875] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.917717 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3826594d300342c7a59e0eee989a873f
I20240503 14:12:01.924042 28622 leader_election.cc:288] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:01.919334 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85b6db1a21c4bca80c8bdec17e45c59" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:01.925336 28290 leader_election.cc:341] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.916765 28661 tablet_replica.cc:330] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:01.925758 28661 raft_consensus.cc:2229] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.926021 28661 raft_consensus.cc:2258] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.926738 28661 ts_tablet_manager.cc:1875] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.930514 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ec6186c8f2e546f39bf0f632d76166fa
I20240503 14:12:01.931203 28032 catalog_manager.cc:6554] ReplaceTablet: tablet ec6186c8f2e546f39bf0f632d76166fa deleted and replaced by tablet dd4c34703a0b46c4a6cf756326602080
I20240503 14:12:01.914846 28653 ts_tablet_manager.cc:1888] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.918460 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85b6db1a21c4bca80c8bdec17e45c59" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:01.946246 28653 log.cc:1196] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e85b6db1a21c4bca80c8bdec17e45c59
I20240503 14:12:01.946784 28653 ts_tablet_manager.cc:1909] T e85b6db1a21c4bca80c8bdec17e45c59 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:01.946848 28290 leader_election.cc:341] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:01.947084 28290 leader_election.cc:302] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.947712 28622 raft_consensus.cc:2726] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.952523 28600 tablet_bootstrap.cc:492] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:01.954370 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 3826594d300342c7a59e0eee989a873f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.955133 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 3826594d300342c7a59e0eee989a873f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.955394 28653 tablet_replica.cc:330] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:01.955705 28653 raft_consensus.cc:2229] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:01.956197 28653 raft_consensus.cc:2258] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.956974 28603 ts_tablet_manager.cc:1888] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.957190 28603 log.cc:1196] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/fd98387c1cae45dcb29c2116b6141708
I20240503 14:12:01.957417 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 3826594d300342c7a59e0eee989a873f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:01.959158 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ae10db451b4e3a8a85d1d987c447d9 with delete_type TABLET_DATA_DELETED (replaced by tablet bacf2d1091ee458c961c691093ea65a9) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.959939 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c6782582fb543f4b621569343995dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.960600 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ec68f75fa2435698383a03874a5bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.961498 28603 ts_tablet_manager.cc:1909] T fd98387c1cae45dcb29c2116b6141708 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:01.961707 28653 ts_tablet_manager.cc:1875] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.967586 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.967854 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.967952 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet f54dc481079c4a5db5b54d485991fa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.968122 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ec68f75fa2435698383a03874a5bae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.968222 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f15be6b9e4baaba9587c3634f72ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:01.968022 28661 ts_tablet_manager.cc:1888] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:01.971925 28653 ts_tablet_manager.cc:1888] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:01.972393 28653 log.cc:1196] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/3826594d300342c7a59e0eee989a873f
I20240503 14:12:01.972995 28661 log.cc:1196] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/67ec68f75fa2435698383a03874a5bae
I20240503 14:12:01.973320 28653 ts_tablet_manager.cc:1909] T 3826594d300342c7a59e0eee989a873f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:01.973798 28661 ts_tablet_manager.cc:1909] T 67ec68f75fa2435698383a03874a5bae P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:01.976652 28603 tablet_replica.cc:330] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:01.977298 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa994dfa54a47408bfb34bcfda9a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:01.977825 28603 raft_consensus.cc:2229] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.978466 28603 raft_consensus.cc:2258] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.979039 28661 tablet_replica.cc:330] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:01.979914 28661 raft_consensus.cc:2229] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:01.979745 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0fa4ad1ec7f348019705a71ba875350d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:01.980644 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fa4ad1ec7f348019705a71ba875350d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.980691 28661 raft_consensus.cc:2258] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:01.980952 28603 ts_tablet_manager.cc:1875] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.981349 28661 ts_tablet_manager.cc:1875] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:01.983471 28610 raft_consensus.cc:482] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.983875 28610 raft_consensus.cc:504] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.984982 28610 leader_election.cc:288] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.985175 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c617bc5bcd1947f38f425f5624ec882c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:01.985913 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c800e997e944158f7c839dd0e00d44" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:01.985913 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c800e997e944158f7c839dd0e00d44" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:01.986027 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c617bc5bcd1947f38f425f5624ec882c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:01.986912 28150 leader_election.cc:341] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:01.987284 28150 leader_election.cc:341] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.987432 28150 leader_election.cc:302] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:01.988209 28610 raft_consensus.cc:2726] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:01.988371 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f3298a69afb440bc9a9c865eb05e070d" } new_table_name: "default.Table_2e3372d35024401296239c262dc5fc5b" modify_external_catalogs: true
I20240503 14:12:01.989518 28634 raft_consensus.cc:482] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:01.989760 28634 raft_consensus.cc:504] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.990732 28634 leader_election.cc:288] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:01.992048 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166fe6366ba496a869af1b106fe9d8e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:01.992327 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166fe6366ba496a869af1b106fe9d8e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:01.992655 28216 raft_consensus.cc:2454] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:01.994123 28427 leader_election.cc:302] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:01.994634 28427 leader_election.cc:341] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:01.995071 28634 raft_consensus.cc:2781] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:01.995244 28634 raft_consensus.cc:482] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:01.995364 28634 raft_consensus.cc:3037] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.999219 28634 raft_consensus.cc:504] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:01.999951 28661 ts_tablet_manager.cc:1888] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.000377 27927 master_runner.cc:186] Error getting master registration for 127.27.50.254:36571: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111),
I20240503 14:12:02.000607 27927 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 14:12:02.001888 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166fe6366ba496a869af1b106fe9d8e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:02.002213 28216 raft_consensus.cc:3037] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.005980 28216 raft_consensus.cc:2454] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:02.007653 28603 ts_tablet_manager.cc:1888] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.007860 28603 log.cc:1196] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e85b6db1a21c4bca80c8bdec17e45c59
I20240503 14:12:02.008396 28603 ts_tablet_manager.cc:1909] T e85b6db1a21c4bca80c8bdec17e45c59 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:02.006839 28427 leader_election.cc:302] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.009557 28603 tablet_replica.cc:330] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:02.009835 28603 raft_consensus.cc:2229] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.010144 28603 raft_consensus.cc:2258] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.011720 28754 raft_consensus.cc:2781] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.011837 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166fe6366ba496a869af1b106fe9d8e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:02.012287 28754 raft_consensus.cc:686] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
W20240503 14:12:02.014385 28427 leader_election.cc:341] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.014942 28634 leader_election.cc:288] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.015748 28603 ts_tablet_manager.cc:1875] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.015575 28754 consensus_queue.cc:237] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.018055 28622 raft_consensus.cc:482] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.018290 28622 raft_consensus.cc:504] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:02.018620 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet a50237f7cc324e61a27ef6fe2dd7f071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.019199 28622 leader_election.cc:288] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.019819 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfdcb8e2b241ca9a6922fd8491e27b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.019820 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfdcb8e2b241ca9a6922fd8491e27b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.020150 28216 raft_consensus.cc:2454] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.020150 28494 raft_consensus.cc:2454] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.020784 28290 leader_election.cc:302] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.020840 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 71598d0a44384a3b93dec30d6c60ecd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.021433 28622 raft_consensus.cc:2781] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.021730 28622 raft_consensus.cc:482] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.021878 28622 raft_consensus.cc:3037] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.025924 28622 raft_consensus.cc:504] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:02.026108 28661 log.cc:1196] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6aa994dfa54a47408bfb34bcfda9a415
I20240503 14:12:02.026867 28661 ts_tablet_manager.cc:1909] T 6aa994dfa54a47408bfb34bcfda9a415 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:02.027792 28622 leader_election.cc:288] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.027810 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfdcb8e2b241ca9a6922fd8491e27b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:02.028239 28216 raft_consensus.cc:3037] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.028231 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfdcb8e2b241ca9a6922fd8491e27b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:02.028477 28494 raft_consensus.cc:3037] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.029481 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6186c8f2e546f39bf0f632d76166fa with delete_type TABLET_DATA_DELETED (replaced by tablet dd4c34703a0b46c4a6cf756326602080) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.029660 28603 ts_tablet_manager.cc:1888] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:02.029848 28603 log.cc:1196] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/3826594d300342c7a59e0eee989a873f
I20240503 14:12:02.030042 28653 tablet_replica.cc:330] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:02.030297 28653 raft_consensus.cc:2229] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.030325 28603 ts_tablet_manager.cc:1909] T 3826594d300342c7a59e0eee989a873f P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:02.030583 28653 raft_consensus.cc:2258] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.031896 28216 raft_consensus.cc:2454] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.032076 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6186c8f2e546f39bf0f632d76166fa with delete_type TABLET_DATA_DELETED (replaced by tablet dd4c34703a0b46c4a6cf756326602080) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.032405 28290 leader_election.cc:302] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:02.032780 28494 raft_consensus.cc:2454] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.032824 28622 raft_consensus.cc:2781] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.033258 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6186c8f2e546f39bf0f632d76166fa with delete_type TABLET_DATA_DELETED (replaced by tablet dd4c34703a0b46c4a6cf756326602080) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.033588 28622 raft_consensus.cc:686] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:02.034068 28622 consensus_queue.cc:237] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:02.034873 28600 tablet_bootstrap.cc:654] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.037312 28614 tablet_bootstrap.cc:492] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.037628 28614 ts_tablet_manager.cc:1367] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.252s user 0.005s sys 0.021s
I20240503 14:12:02.038857 28614 raft_consensus.cc:348] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.039961 28614 raft_consensus.cc:374] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.040375 28614 raft_consensus.cc:729] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.042243 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 38c0611cf7aa4ff3b4e7e6081558729b (table default.Table_ae2a19fbff3449aca0cc7c9cd3e01e1e [id=e6b422e744c14453a7e7c4b28c1eaac1]) successfully deleted
I20240503 14:12:02.042706 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 67ec68f75fa2435698383a03874a5bae (table default.Table_a9c4dd16181e4b0789779c3f25eaed1c [id=18925bb0895246988de55da2f108454b]) successfully deleted
I20240503 14:12:02.042354 28614 consensus_queue.cc:260] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.045781 28653 ts_tablet_manager.cc:1875] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.046020 28661 tablet_replica.cc:330] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:02.046274 28661 raft_consensus.cc:2229] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.046538 28661 raft_consensus.cc:2258] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.047514 28661 ts_tablet_manager.cc:1875] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.048875 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet fd98387c1cae45dcb29c2116b6141708 (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]) successfully deleted
W20240503 14:12:02.051745 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2502a13e8cbc4a3abfed225f94c44d2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2502a13e8cbc4a3abfed225f94c44d2f already in progress: creating tablet
I20240503 14:12:02.052131 28494 raft_consensus.cc:1261] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Refusing update from remote peer d3a6b9b7913444d29c96581af6febf33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:02.052198 28216 raft_consensus.cc:1261] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Refusing update from remote peer d3a6b9b7913444d29c96581af6febf33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:02.053150 28715 consensus_queue.cc:1035] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:02.053556 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ab05178dc081480ba409aab7f143f849 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab05178dc081480ba409aab7f143f849 already in progress: creating tablet
W20240503 14:12:02.053817 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 559342d357d64f9eb9e3ab42ccb75942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 559342d357d64f9eb9e3ab42ccb75942 already in progress: creating tablet
I20240503 14:12:02.058494 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e85b6db1a21c4bca80c8bdec17e45c59 (table default.Table_7afc535ef1bb440493a9e65dc3c86fde [id=52c4ea4489f24aa19d1671b4a5f88808]) successfully deleted
I20240503 14:12:02.058836 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 3826594d300342c7a59e0eee989a873f (table default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b]) successfully deleted
I20240503 14:12:02.059084 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e85b6db1a21c4bca80c8bdec17e45c59 (table default.Table_7afc535ef1bb440493a9e65dc3c86fde [id=52c4ea4489f24aa19d1671b4a5f88808]) successfully deleted
W20240503 14:12:02.059397 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 559342d357d64f9eb9e3ab42ccb75942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 559342d357d64f9eb9e3ab42ccb75942 already in progress: creating tablet
I20240503 14:12:02.059634 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 3826594d300342c7a59e0eee989a873f (table default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b]) successfully deleted
W20240503 14:12:02.059906 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6aa994dfa54a47408bfb34bcfda9a415 because the tablet was not found. No further retry: Not found: Tablet not found: 6aa994dfa54a47408bfb34bcfda9a415
W20240503 14:12:02.060164 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f9713865e09349ce9f9ae1d4297da2d0 because the tablet was not found. No further retry: Not found: Tablet not found: f9713865e09349ce9f9ae1d4297da2d0
W20240503 14:12:02.060416 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f54dc481079c4a5db5b54d485991fa65 because the tablet was not found. No further retry: Not found: Tablet not found: f54dc481079c4a5db5b54d485991fa65
I20240503 14:12:02.060885 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6aa994dfa54a47408bfb34bcfda9a415 (table default.Table_4e83fe8e46644b8f8df871f6da4f0f1d [id=a1a30b8a5adb44b390e6bea2dd82c20c]) successfully deleted
W20240503 14:12:02.061163 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ab05178dc081480ba409aab7f143f849 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab05178dc081480ba409aab7f143f849 already in progress: creating tablet
W20240503 14:12:02.061395 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 559342d357d64f9eb9e3ab42ccb75942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 559342d357d64f9eb9e3ab42ccb75942 already in progress: creating tablet
W20240503 14:12:02.061657 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e85b6db1a21c4bca80c8bdec17e45c59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e85b6db1a21c4bca80c8bdec17e45c59 already in progress: creating tablet
I20240503 14:12:02.062022 28614 ts_tablet_manager.cc:1398] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240503 14:12:02.064033 28715 consensus_queue.cc:1035] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240503 14:12:02.066452 28603 tablet_replica.cc:330] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:02.071938 28603 raft_consensus.cc:2229] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.072409 28603 raft_consensus.cc:2258] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.072983 28600 tablet_bootstrap.cc:492] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.073483 28600 ts_tablet_manager.cc:1367] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.001s
I20240503 14:12:02.073674 28603 ts_tablet_manager.cc:1875] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.075469 28600 raft_consensus.cc:348] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.076128 28600 raft_consensus.cc:374] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.076403 28600 raft_consensus.cc:729] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.076747 28614 tablet_bootstrap.cc:492] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.077215 28600 consensus_queue.cc:260] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.078977 28614 tablet_bootstrap.cc:654] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.079578 28600 ts_tablet_manager.cc:1398] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240503 14:12:02.080690 28600 tablet_bootstrap.cc:492] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.086134 28600 tablet_bootstrap.cc:654] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.091365 28705 tablet.cc:1722] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:02.091768 28711 tablet.cc:1722] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:02.096315 28662 raft_consensus.cc:482] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.096535 28662 raft_consensus.cc:504] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.097364 28662 leader_election.cc:288] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.098490 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe5d280a1654d80a2176c1def459e2e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.098776 28217 raft_consensus.cc:2454] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.100378 28290 leader_election.cc:302] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:02.100843 28662 raft_consensus.cc:2781] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.101025 28662 raft_consensus.cc:482] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.101156 28662 raft_consensus.cc:3037] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.102747 28604 tablet_bootstrap.cc:492] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.103004 28604 ts_tablet_manager.cc:1367] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.208s user 0.025s sys 0.061s
I20240503 14:12:02.103781 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe5d280a1654d80a2176c1def459e2e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.103984 28604 raft_consensus.cc:348] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.104271 28604 raft_consensus.cc:374] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.104400 28604 raft_consensus.cc:729] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:02.104743 28290 leader_election.cc:341] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.104694 28604 consensus_queue.cc:260] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.106091 28610 raft_consensus.cc:482] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.106307 28604 ts_tablet_manager.cc:1398] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:02.106302 28610 raft_consensus.cc:504] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.106887 28604 tablet_bootstrap.cc:492] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.107139 28610 leader_election.cc:288] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.107753 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacf2d1091ee458c961c691093ea65a9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.108371 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacf2d1091ee458c961c691093ea65a9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:02.108446 28150 leader_election.cc:341] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:02.108916 28150 leader_election.cc:341] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.109138 28150 leader_election.cc:302] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:02.109476 28610 raft_consensus.cc:2726] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.109622 28661 ts_tablet_manager.cc:1888] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.109820 28661 log.cc:1196] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/f54dc481079c4a5db5b54d485991fa65
I20240503 14:12:02.111251 28662 raft_consensus.cc:504] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.111439 28661 ts_tablet_manager.cc:1909] T f54dc481079c4a5db5b54d485991fa65 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:02.111636 28604 tablet_bootstrap.cc:654] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.112332 28662 leader_election.cc:288] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.112826 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe5d280a1654d80a2176c1def459e2e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:02.112860 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe5d280a1654d80a2176c1def459e2e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:02.113142 28217 raft_consensus.cc:3037] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:02.113595 28290 leader_election.cc:341] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.113983 28705 tablet.cc:1717] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:02.116341 28711 tablet.cc:1717] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:02.116730 28217 raft_consensus.cc:2454] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.120694 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet f54dc481079c4a5db5b54d485991fa65 (table default.Table_9289210d002542b78ccb48780cf16018 [id=3e4bfd54a342491b86e978c08faa5c8b]) successfully deleted
I20240503 14:12:02.123296 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_870eff51fbc54c9187d45df3f9cbb08f" } new_table_name: "default.Table_7d209c20164a474485dae171a960b5f4" modify_external_catalogs: true
I20240503 14:12:02.125969 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_de96804c306a4b0e91a2e6bffd5943a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.126710 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de96804c306a4b0e91a2e6bffd5943a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.127133 28604 tablet_bootstrap.cc:492] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.127425 28604 ts_tablet_manager.cc:1367] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20240503 14:12:02.128466 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8a55dd8eb2b449d0bf197060444b3571" } new_table_name: "default.Table_2ad02256d653464e9528725b5e37c72c" modify_external_catalogs: true
I20240503 14:12:02.128536 28604 raft_consensus.cc:348] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.128825 28604 raft_consensus.cc:374] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.128933 28604 raft_consensus.cc:729] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.129272 28604 consensus_queue.cc:260] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.129889 28604 ts_tablet_manager.cc:1398] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.130299 28604 tablet_bootstrap.cc:492] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.137133 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]). Expected version 3 got 0
I20240503 14:12:02.137733 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 7fe5d280a1654d80a2176c1def459e2e (table default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]). Expected version 1 got 0
I20240503 14:12:02.137971 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_456be1a488a44191be1e21de223834c9 [id=548632b6d08e4435ad561b074e5e5b14]). Expected version 1 got 0
I20240503 14:12:02.138401 28290 leader_election.cc:302] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:02.138962 28662 raft_consensus.cc:2781] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.139204 28662 raft_consensus.cc:686] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:02.139566 28662 consensus_queue.cc:237] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.140370 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 7fe5d280a1654d80a2176c1def459e2e (table default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]). Expected version 1 got 0
I20240503 14:12:02.140612 28604 tablet_bootstrap.cc:654] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.140672 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]). Expected version 3 got 0
I20240503 14:12:02.141764 28031 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_456be1a488a44191be1e21de223834c9 [id=548632b6d08e4435ad561b074e5e5b14]). Expected version 1 got 0
I20240503 14:12:02.142040 28031 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
I20240503 14:12:02.148938 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_41714410a87e4a7cacd1dd22125c433f" } new_table_name: "default.Table_2ef7a9ee753245dcad1ff57366d3eb99" modify_external_catalogs: true
I20240503 14:12:02.151703 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet b1de9717d0e44179b875174d5a76e4c3 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:02.153726 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ef6cd66f34c4980bad5e500309b1568 consensus state has no leader
I20240503 14:12:02.153810 28614 tablet_bootstrap.cc:492] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.154120 28614 ts_tablet_manager.cc:1367] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.003s
I20240503 14:12:02.154206 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d637087c60144aaaa9b0965110d280e from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:02.164772 28614 raft_consensus.cc:348] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.166721 28614 raft_consensus.cc:374] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.167074 28661 tablet_replica.cc:330] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:02.167280 28614 raft_consensus.cc:729] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.167732 28661 raft_consensus.cc:2229] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.168040 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 1751f775fe6943d9b5fcc3b113393a73 (DEFAULT_TABLE table=default.Table_d56c053e47bb4621b5a7d74ca45069cc [id=9931742257d74df99c043e569a267ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.168757 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1751f775fe6943d9b5fcc3b113393a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.088032 28603 ts_tablet_manager.cc:1888] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.169229 28603 log.cc:1196] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/87ae10db451b4e3a8a85d1d987c447d9
I20240503 14:12:02.175055 28692 tablet.cc:1722] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:02.171664 28754 raft_consensus.cc:482] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.175824 28754 raft_consensus.cc:504] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.176800 28754 leader_election.cc:288] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:02.180235 28427 leader_election.cc:341] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: f35e87db1c2a42a79b426b2e7767367a
W20240503 14:12:02.180768 28427 leader_election.cc:341] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: f35e87db1c2a42a79b426b2e7767367a
I20240503 14:12:02.181087 28427 leader_election.cc:302] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:02.181802 28754 raft_consensus.cc:2726] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.185359 28661 raft_consensus.cc:2258] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.185923 28603 ts_tablet_manager.cc:1909] T 87ae10db451b4e3a8a85d1d987c447d9 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:02.187058 28661 ts_tablet_manager.cc:1875] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.173566 28614 consensus_queue.cc:260] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.190595 28614 ts_tablet_manager.cc:1398] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.036s user 0.002s sys 0.002s
I20240503 14:12:02.192782 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 4775ff1029e6449c83219030e3809984 (DEFAULT_TABLE table=default.Table_8a55dd8eb2b449d0bf197060444b3571 [id=36785e97f5c94a15bf717d499a33f558]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.191768 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 1751f775fe6943d9b5fcc3b113393a73 (DEFAULT_TABLE table=default.Table_d56c053e47bb4621b5a7d74ca45069cc [id=9931742257d74df99c043e569a267ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.194264 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1751f775fe6943d9b5fcc3b113393a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.194639 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 (DEFAULT_TABLE table=default.Table_6d7feb39cce84e66b75914a6bcbcfcc9 [id=56b86fbb2fff40cfa3729f88092d57ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.193663 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 4775ff1029e6449c83219030e3809984 (DEFAULT_TABLE table=default.Table_8a55dd8eb2b449d0bf197060444b3571 [id=36785e97f5c94a15bf717d499a33f558]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.196738 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4775ff1029e6449c83219030e3809984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.197335 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 (DEFAULT_TABLE table=default.Table_6d7feb39cce84e66b75914a6bcbcfcc9 [id=56b86fbb2fff40cfa3729f88092d57ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.197798 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbabe5beb9a42c69cb64e1b364d75a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.198881 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7d57fc4486457e9848daad79f04356 (DEFAULT_TABLE table=default.Table_4dd04c06a7a04769b9b6eef8faac57a1 [id=5cf73f88a73a4cc5966467ecfff2140f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.190627 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 1751f775fe6943d9b5fcc3b113393a73 (DEFAULT_TABLE table=default.Table_d56c053e47bb4621b5a7d74ca45069cc [id=9931742257d74df99c043e569a267ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.200676 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7d57fc4486457e9848daad79f04356 (DEFAULT_TABLE table=default.Table_4dd04c06a7a04769b9b6eef8faac57a1 [id=5cf73f88a73a4cc5966467ecfff2140f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.191916 28604 tablet_bootstrap.cc:492] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.201192 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1751f775fe6943d9b5fcc3b113393a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.201480 28197 tablet_service.cc:1467] Processing CreateTablet for tablet ea3e9c030a424740bb2fb641328adfe1 (DEFAULT_TABLE table=default.Table_baa3d7860d0b469899a33511f6bb9a29 [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.192567 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 4775ff1029e6449c83219030e3809984 (DEFAULT_TABLE table=default.Table_8a55dd8eb2b449d0bf197060444b3571 [id=36785e97f5c94a15bf717d499a33f558]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.196280 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 (DEFAULT_TABLE table=default.Table_6d7feb39cce84e66b75914a6bcbcfcc9 [id=56b86fbb2fff40cfa3729f88092d57ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.201535 28604 ts_tablet_manager.cc:1367] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.014s
I20240503 14:12:02.202978 28614 tablet_bootstrap.cc:492] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.202987 28600 tablet_bootstrap.cc:492] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.203166 28600 ts_tablet_manager.cc:1367] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.021s
I20240503 14:12:02.203583 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4775ff1029e6449c83219030e3809984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.203886 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbabe5beb9a42c69cb64e1b364d75a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.204253 28600 raft_consensus.cc:348] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.204514 28600 raft_consensus.cc:374] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.204653 28600 raft_consensus.cc:729] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.207144 28603 tablet_replica.cc:330] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:02.207393 28600 consensus_queue.cc:260] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.207763 28467 tablet_service.cc:1467] Processing CreateTablet for tablet ea3e9c030a424740bb2fb641328adfe1 (DEFAULT_TABLE table=default.Table_baa3d7860d0b469899a33511f6bb9a29 [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.208123 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7d57fc4486457e9848daad79f04356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.210152 28600 ts_tablet_manager.cc:1398] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:02.211151 28600 tablet_bootstrap.cc:492] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.213591 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7d57fc4486457e9848daad79f04356 (DEFAULT_TABLE table=default.Table_4dd04c06a7a04769b9b6eef8faac57a1 [id=5cf73f88a73a4cc5966467ecfff2140f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.214704 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7d57fc4486457e9848daad79f04356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.214112 28600 tablet_bootstrap.cc:654] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.219959 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3e9c030a424740bb2fb641328adfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.226889 28603 raft_consensus.cc:2229] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.229190 28604 raft_consensus.cc:348] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.229485 28604 raft_consensus.cc:374] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.229655 28604 raft_consensus.cc:729] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.230029 28604 consensus_queue.cc:260] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.231444 28604 ts_tablet_manager.cc:1398] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.028s user 0.001s sys 0.002s
I20240503 14:12:02.232102 28614 tablet_bootstrap.cc:654] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.232398 28604 tablet_bootstrap.cc:492] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.232318 28192 tablet_service.cc:1467] Processing CreateTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 (DEFAULT_TABLE table=default.Table_22032e3385624e98a1b0bb1484f51c64 [id=304746db399b4284999a5ca79e9e80f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.234104 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 1693dd36a3d84d88a49690c537590083 (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.235594 28194 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c (DEFAULT_TABLE table=default.Table_4033322e04a3488cbc8be0fe6a61bec1 [id=e3f646d9aa8946529e897c83c0b10f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.236989 28193 tablet_service.cc:1467] Processing CreateTablet for tablet da19787034c24ebe929601d49cb21b3b (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.237530 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da19787034c24ebe929601d49cb21b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.237972 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4775ff1029e6449c83219030e3809984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.238234 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1693dd36a3d84d88a49690c537590083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.238634 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d2600031df4d77b9afe636a9b5ad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.238862 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbabe5beb9a42c69cb64e1b364d75a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.239504 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3e9c030a424740bb2fb641328adfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.239418 28469 tablet_service.cc:1467] Processing CreateTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 (DEFAULT_TABLE table=default.Table_22032e3385624e98a1b0bb1484f51c64 [id=304746db399b4284999a5ca79e9e80f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.239902 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d2600031df4d77b9afe636a9b5ad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.240111 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7d57fc4486457e9848daad79f04356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.240885 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.241972 28603 raft_consensus.cc:2258] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.243561 28692 tablet.cc:1717] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:02.243559 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 1693dd36a3d84d88a49690c537590083 (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.244148 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1693dd36a3d84d88a49690c537590083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.245477 28470 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c (DEFAULT_TABLE table=default.Table_4033322e04a3488cbc8be0fe6a61bec1 [id=e3f646d9aa8946529e897c83c0b10f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.245965 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.247716 28754 raft_consensus.cc:482] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.247961 28754 raft_consensus.cc:504] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.256028 28604 tablet_bootstrap.cc:654] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.256383 28603 ts_tablet_manager.cc:1875] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:02.258040 28427 leader_election.cc:341] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: f9713865e09349ce9f9ae1d4297da2d0
I20240503 14:12:02.256045 28754 leader_election.cc:288] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.258625 28662 raft_consensus.cc:482] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.258788 28195 tablet_service.cc:1467] Processing CreateTablet for tablet e355aafb2f5b4337bedaac6fe5539dc2 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.258821 28188 tablet_service.cc:1467] Processing CreateTablet for tablet eee989ff083041dc8dcd2cd1b220780d (DEFAULT_TABLE table=default.Table_706c4da08fa84bbaa207fc71eb482682 [id=36f9af99d9d44c32876688fc8cbc4f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.260361 28662 raft_consensus.cc:504] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.260568 28194 tablet_service.cc:1467] Processing CreateTablet for tablet fde0e59d911d4c1eb5125b1f869b396c (DEFAULT_TABLE table=default.Table_870eff51fbc54c9187d45df3f9cbb08f [id=a66a34594c864f0c81ba2da0a1449aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.261358 28662 leader_election.cc:288] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.259225 28465 tablet_service.cc:1467] Processing CreateTablet for tablet e355aafb2f5b4337bedaac6fe5539dc2 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.263427 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 87ae10db451b4e3a8a85d1d987c447d9 (table default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]) successfully deleted
W20240503 14:12:02.263881 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 4c6782582fb543f4b621569343995dfc because the tablet was not found. No further retry: Not found: Tablet not found: 4c6782582fb543f4b621569343995dfc
I20240503 14:12:02.263832 28467 tablet_service.cc:1467] Processing CreateTablet for tablet da19787034c24ebe929601d49cb21b3b (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:02.264194 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 67ec68f75fa2435698383a03874a5bae because the tablet was not found. No further retry: Not found: Tablet not found: 67ec68f75fa2435698383a03874a5bae
I20240503 14:12:02.268926 28336 tablet_service.cc:1467] Processing CreateTablet for tablet ea3e9c030a424740bb2fb641328adfe1 (DEFAULT_TABLE table=default.Table_baa3d7860d0b469899a33511f6bb9a29 [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.269908 28330 tablet_service.cc:1467] Processing CreateTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 (DEFAULT_TABLE table=default.Table_22032e3385624e98a1b0bb1484f51c64 [id=304746db399b4284999a5ca79e9e80f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.271658 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 1693dd36a3d84d88a49690c537590083 (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.271893 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da19787034c24ebe929601d49cb21b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.271307 28464 tablet_service.cc:1467] Processing CreateTablet for tablet eee989ff083041dc8dcd2cd1b220780d (DEFAULT_TABLE table=default.Table_706c4da08fa84bbaa207fc71eb482682 [id=36f9af99d9d44c32876688fc8cbc4f8b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:02.279094 28290 leader_election.cc:341] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6106d52df3c54c2bb4abadd8605cd912
I20240503 14:12:02.279138 28329 tablet_service.cc:1467] Processing CreateTablet for tablet fde0e59d911d4c1eb5125b1f869b396c (DEFAULT_TABLE table=default.Table_870eff51fbc54c9187d45df3f9cbb08f [id=a66a34594c864f0c81ba2da0a1449aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.281062 28331 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c (DEFAULT_TABLE table=default.Table_4033322e04a3488cbc8be0fe6a61bec1 [id=e3f646d9aa8946529e897c83c0b10f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.282063 28337 tablet_service.cc:1467] Processing CreateTablet for tablet e355aafb2f5b4337bedaac6fe5539dc2 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.282986 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d2600031df4d77b9afe636a9b5ad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.283098 28662 raft_consensus.cc:482] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.283665 28600 tablet_bootstrap.cc:492] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.283977 28600 ts_tablet_manager.cc:1367] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.027s
I20240503 14:12:02.285108 28662 raft_consensus.cc:504] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.286034 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e355aafb2f5b4337bedaac6fe5539dc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:02.286725 28427 leader_election.cc:341] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: f9713865e09349ce9f9ae1d4297da2d0
I20240503 14:12:02.286693 28600 raft_consensus.cc:348] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.287362 28600 raft_consensus.cc:374] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.287443 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6106d52df3c54c2bb4abadd8605cd912" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.287648 28600 raft_consensus.cc:729] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.288487 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba629915ef44e0882beee2612e285c1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.288415 28600 consensus_queue.cc:260] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.288791 28216 raft_consensus.cc:2454] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.289780 28494 raft_consensus.cc:2454] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.289742 28334 tablet_service.cc:1467] Processing CreateTablet for tablet da19787034c24ebe929601d49cb21b3b (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.290530 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba629915ef44e0882beee2612e285c1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.290752 28332 tablet_service.cc:1467] Processing CreateTablet for tablet eee989ff083041dc8dcd2cd1b220780d (DEFAULT_TABLE table=default.Table_706c4da08fa84bbaa207fc71eb482682 [id=36f9af99d9d44c32876688fc8cbc4f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.290813 28492 raft_consensus.cc:2454] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.291399 28427 leader_election.cc:302] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:02.291697 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3e9c030a424740bb2fb641328adfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.293437 28600 ts_tablet_manager.cc:1398] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240503 14:12:02.294775 28600 tablet_bootstrap.cc:492] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.296020 28754 raft_consensus.cc:2726] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.296810 28662 leader_election.cc:288] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.297135 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee989ff083041dc8dcd2cd1b220780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.297510 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e355aafb2f5b4337bedaac6fe5539dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.299095 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da19787034c24ebe929601d49cb21b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.299114 28290 leader_election.cc:302] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:02.299736 28290 leader_election.cc:302] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:02.301010 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde0e59d911d4c1eb5125b1f869b396c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.303218 28662 raft_consensus.cc:2781] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.303226 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.303655 28662 raft_consensus.cc:482] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.303797 28662 raft_consensus.cc:3037] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.284325 28466 tablet_service.cc:1467] Processing CreateTablet for tablet fde0e59d911d4c1eb5125b1f869b396c (DEFAULT_TABLE table=default.Table_870eff51fbc54c9187d45df3f9cbb08f [id=a66a34594c864f0c81ba2da0a1449aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.311221 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1693dd36a3d84d88a49690c537590083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.311504 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee989ff083041dc8dcd2cd1b220780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.310734 28754 raft_consensus.cc:482] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.312369 28715 raft_consensus.cc:2781] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.312558 28715 raft_consensus.cc:482] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.312404 28754 raft_consensus.cc:504] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.312670 28715 raft_consensus.cc:3037] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.313175 28754 leader_election.cc:288] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.313894 28614 tablet_bootstrap.cc:492] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.314118 28614 ts_tablet_manager.cc:1367] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.114s user 0.008s sys 0.003s
I20240503 14:12:02.314628 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe5d280a1654d80a2176c1def459e2e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.314937 28216 raft_consensus.cc:2379] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:02.315169 28614 raft_consensus.cc:348] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.315444 28614 raft_consensus.cc:374] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.315601 28614 raft_consensus.cc:729] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.316270 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe5d280a1654d80a2176c1def459e2e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.316272 28614 consensus_queue.cc:260] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.319571 28427 leader_election.cc:302] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:02.320793 28754 raft_consensus.cc:3037] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.322027 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b1de9717d0e44179b875174d5a76e4c3
I20240503 14:12:02.322645 28026 catalog_manager.cc:6554] ReplaceTablet: tablet b1de9717d0e44179b875174d5a76e4c3 deleted and replaced by tablet a32b6995818041778617a0d6bf936c8c
I20240503 14:12:02.323009 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 7fe5d280a1654d80a2176c1def459e2e from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:02.323239 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fde0e59d911d4c1eb5125b1f869b396c consensus state has no leader
I20240503 14:12:02.323870 28600 tablet_bootstrap.cc:654] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.323793 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3dd432e943764d9296fada2cf3c0ca47" } modify_external_catalogs: true
I20240503 14:12:02.324035 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3dd432e943764d9296fada2cf3c0ca47" } modify_external_catalogs: true
I20240503 14:12:02.325860 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 6aa994dfa54a47408bfb34bcfda9a415 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:02.326571 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_baa3d7860d0b469899a33511f6bb9a29" } new_table_name: "default.Table_b85675e32b564761a3df4d8b9a1cbc45" modify_external_catalogs: true
W20240503 14:12:02.328223 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4775ff1029e6449c83219030e3809984 consensus state has no leader
I20240503 14:12:02.328706 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a07e2f0650084270860bd28e40f224aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.329394 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a07e2f0650084270860bd28e40f224aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.330852 28030 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_f2e8bc9e82a64d6682fa813e04fe7fc4 [id=3a24797cc53b439da44cc77025fa9f92] alter complete (version 4)
I20240503 14:12:02.332438 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3c2d3cac1d8c4aa4b9730b7f0baa45a0" } modify_external_catalogs: true
I20240503 14:12:02.332675 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3c2d3cac1d8c4aa4b9730b7f0baa45a0" } modify_external_catalogs: true
I20240503 14:12:02.334985 28600 tablet_bootstrap.cc:492] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.335063 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a0e8b0eed57c4ad1bf6040507f6f8d4c
I20240503 14:12:02.335249 28600 ts_tablet_manager.cc:1367] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.006s
I20240503 14:12:02.336278 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:02.336485 28600 raft_consensus.cc:348] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.336776 28600 raft_consensus.cc:374] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.336999 28604 tablet_bootstrap.cc:492] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.337011 28600 raft_consensus.cc:729] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.337296 28604 ts_tablet_manager.cc:1367] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.004s
I20240503 14:12:02.337298 28600 consensus_queue.cc:260] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.339450 28600 ts_tablet_manager.cc:1398] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:02.339843 28600 tablet_bootstrap.cc:492] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.341053 28610 raft_consensus.cc:482] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.342133 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde0e59d911d4c1eb5125b1f869b396c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.343837 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_59af1c61c0534502b0f2cdaff98f4e96" } modify_external_catalogs: true
I20240503 14:12:02.345216 28662 raft_consensus.cc:504] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.346060 28662 leader_election.cc:288] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.338296 28604 raft_consensus.cc:348] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.347777 28604 raft_consensus.cc:374] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.347924 28604 raft_consensus.cc:729] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.348351 28604 consensus_queue.cc:260] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.349155 28604 ts_tablet_manager.cc:1398] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 14:12:02.349615 28604 tablet_bootstrap.cc:492] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.352342 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e355aafb2f5b4337bedaac6fe5539dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.353308 28604 tablet_bootstrap.cc:654] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.355760 28653 ts_tablet_manager.cc:1888] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.355952 28653 log.cc:1196] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/67ec68f75fa2435698383a03874a5bae
I20240503 14:12:02.356277 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_59af1c61c0534502b0f2cdaff98f4e96" } modify_external_catalogs: true
I20240503 14:12:02.356837 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5d637087c60144aaaa9b0965110d280e
I20240503 14:12:02.341241 28610 raft_consensus.cc:504] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.358163 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 5d637087c60144aaaa9b0965110d280e deleted and replaced by tablet e9609112657f427cbeeb3ef8f28cddf0
I20240503 14:12:02.358667 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_986d778bcd814c9c87dbd18d1f485cc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:02.360412 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_706c4da08fa84bbaa207fc71eb482682" } modify_external_catalogs: true
I20240503 14:12:02.360849 28610 leader_election.cc:288] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.361214 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab05178dc081480ba409aab7f143f849" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.361311 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab05178dc081480ba409aab7f143f849" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.363574 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee989ff083041dc8dcd2cd1b220780d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:02.373497 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_986d778bcd814c9c87dbd18d1f485cc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.376266 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_706c4da08fa84bbaa207fc71eb482682" } modify_external_catalogs: true
W20240503 14:12:02.380738 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.383927 28754 raft_consensus.cc:2726] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.383806 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c2d18c6af36b4d0096425aa0afc52fd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.384660 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2d18c6af36b4d0096425aa0afc52fd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:02.385277 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 68794.1us
I20240503 14:12:02.386219 28332 tablet_service.cc:1514] Processing DeleteTablet for tablet b1de9717d0e44179b875174d5a76e4c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a32b6995818041778617a0d6bf936c8c) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.386278 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba629915ef44e0882beee2612e285c1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:02.386559 28614 ts_tablet_manager.cc:1398] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.072s user 0.003s sys 0.000s
I20240503 14:12:02.386577 28492 raft_consensus.cc:3037] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.387461 28332 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.389025 28614 tablet_bootstrap.cc:492] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.389802 28610 raft_consensus.cc:482] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.390905 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet b1de9717d0e44179b875174d5a76e4c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a32b6995818041778617a0d6bf936c8c) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.391304 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.391647 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet f9713865e09349ce9f9ae1d4297da2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.395565 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:02.395782 28150 leader_election.cc:341] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.396274 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.396823 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d637087c60144aaaa9b0965110d280e with delete_type TABLET_DATA_DELETED (replaced by tablet e9609112657f427cbeeb3ef8f28cddf0) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.399394 28614 tablet_bootstrap.cc:654] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.402738 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde0e59d911d4c1eb5125b1f869b396c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:02.408252 28150 leader_election.cc:341] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.408457 28150 leader_election.cc:302] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:02.416327 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet eb79edba14b646ef9be239b05a291d71
I20240503 14:12:02.416422 28653 ts_tablet_manager.cc:1909] T 67ec68f75fa2435698383a03874a5bae P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:02.416891 28604 tablet_bootstrap.cc:492] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.417098 28604 ts_tablet_manager.cc:1367] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.068s user 0.013s sys 0.002s
I20240503 14:12:02.417716 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba629915ef44e0882beee2612e285c1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:02.418013 28216 raft_consensus.cc:3037] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.418236 28604 raft_consensus.cc:348] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.418521 28604 raft_consensus.cc:374] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.418718 28604 raft_consensus.cc:729] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.419023 28604 consensus_queue.cc:260] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:02.420042 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea3e9c030a424740bb2fb641328adfe1 consensus state has no leader
I20240503 14:12:02.420773 28604 ts_tablet_manager.cc:1398] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:02.421289 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d637087c60144aaaa9b0965110d280e with delete_type TABLET_DATA_DELETED (replaced by tablet e9609112657f427cbeeb3ef8f28cddf0) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.427295 28705 tablet.cc:1722] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:02.430408 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c800e997e944158f7c839dd0e00d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.429227 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d637087c60144aaaa9b0965110d280e with delete_type TABLET_DATA_DELETED (replaced by tablet e9609112657f427cbeeb3ef8f28cddf0) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.413218 28610 raft_consensus.cc:504] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.431443 28661 ts_tablet_manager.cc:1888] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.431970 28603 ts_tablet_manager.cc:1888] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.432113 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.432210 28603 log.cc:1196] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6aa994dfa54a47408bfb34bcfda9a415
I20240503 14:12:02.432181 28216 raft_consensus.cc:2454] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.432585 28603 ts_tablet_manager.cc:1909] T 6aa994dfa54a47408bfb34bcfda9a415 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:02.432655 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.433187 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559342d357d64f9eb9e3ab42ccb75942" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.433929 28653 tablet_replica.cc:330] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:02.434063 28150 leader_election.cc:341] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.434227 28653 raft_consensus.cc:2229] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.434440 28653 raft_consensus.cc:2258] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.435097 28653 ts_tablet_manager.cc:1875] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.436754 28661 log.cc:1196] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a50237f7cc324e61a27ef6fe2dd7f071
I20240503 14:12:02.437666 28661 ts_tablet_manager.cc:1909] T a50237f7cc324e61a27ef6fe2dd7f071 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:02.438014 28610 leader_election.cc:288] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.438913 28661 tablet_replica.cc:330] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:02.439163 28661 raft_consensus.cc:2229] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.439359 28711 tablet.cc:1722] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:02.439906 28610 raft_consensus.cc:2726] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.440109 28661 raft_consensus.cc:2258] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.440778 28661 ts_tablet_manager.cc:1875] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.442250 28492 raft_consensus.cc:2454] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.442792 28604 tablet_bootstrap.cc:492] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.443136 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559342d357d64f9eb9e3ab42ccb75942" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:02.444048 28150 leader_election.cc:341] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.444010 28715 raft_consensus.cc:504] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.445019 28150 leader_election.cc:302] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:02.445644 28610 raft_consensus.cc:2726] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.445943 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 67ec68f75fa2435698383a03874a5bae (table default.Table_a9c4dd16181e4b0789779c3f25eaed1c [id=18925bb0895246988de55da2f108454b]) successfully deleted
I20240503 14:12:02.446379 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet eb79edba14b646ef9be239b05a291d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.446955 28604 tablet_bootstrap.cc:654] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.449612 28600 tablet_bootstrap.cc:654] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.450531 28715 leader_election.cc:288] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:02.451025 28614 tablet_bootstrap.cc:492] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.453786 28614 ts_tablet_manager.cc:1367] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.016s
I20240503 14:12:02.455420 28614 raft_consensus.cc:348] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.456389 28614 raft_consensus.cc:374] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.456542 28614 raft_consensus.cc:729] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.457253 28614 consensus_queue.cc:260] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.458612 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet eb79edba14b646ef9be239b05a291d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.459347 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a50237f7cc324e61a27ef6fe2dd7f071 (table default.Table_0515e83b3d914915a67f1d30f123eb09 [id=f699bab9cb6a4364a6b371df69e16222]) successfully deleted
I20240503 14:12:02.460676 28614 ts_tablet_manager.cc:1398] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:02.463762 28614 tablet_bootstrap.cc:492] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.468626 28614 tablet_bootstrap.cc:654] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.458763 28604 tablet_bootstrap.cc:492] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.469635 28754 raft_consensus.cc:482] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.472846 28604 ts_tablet_manager.cc:1367] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.006s
I20240503 14:12:02.472754 28754 raft_consensus.cc:504] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.474398 28754 leader_election.cc:288] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.475694 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb79edba14b646ef9be239b05a291d71" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.476258 28216 raft_consensus.cc:2454] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:02.476994 28604 raft_consensus.cc:348] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.477108 28427 leader_election.cc:302] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.477488 28754 raft_consensus.cc:2781] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.477641 28754 raft_consensus.cc:482] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.477756 28754 raft_consensus.cc:3037] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.481061 28604 raft_consensus.cc:374] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.481232 28604 raft_consensus.cc:729] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.481609 28604 consensus_queue.cc:260] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.482312 28604 ts_tablet_manager.cc:1398] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:12:02.482714 28604 tablet_bootstrap.cc:492] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.483783 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6106d52df3c54c2bb4abadd8605cd912" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:02.484053 28494 raft_consensus.cc:3037] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.485613 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb79edba14b646ef9be239b05a291d71" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.485800 28290 leader_election.cc:302] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.486199 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet eb79edba14b646ef9be239b05a291d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.486642 28622 raft_consensus.cc:2781] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.486873 28622 raft_consensus.cc:686] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:02.487185 28622 consensus_queue.cc:237] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.489717 28604 tablet_bootstrap.cc:654] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:02.495424 28290 leader_election.cc:341] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6106d52df3c54c2bb4abadd8605cd912
I20240503 14:12:02.500525 28781 raft_consensus.cc:482] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.500973 28781 raft_consensus.cc:504] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.501765 28614 tablet_bootstrap.cc:492] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.501972 28614 ts_tablet_manager.cc:1367] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.004s
I20240503 14:12:02.503923 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c875ec8c5e49d1bb20a23dd7a21902" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.504174 28614 raft_consensus.cc:348] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.504432 28614 raft_consensus.cc:374] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.504441 28692 tablet.cc:1722] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240503 14:12:02.504978 28150 leader_election.cc:341] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.500017 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet b1de9717d0e44179b875174d5a76e4c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a32b6995818041778617a0d6bf936c8c) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.506168 28614 raft_consensus.cc:729] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.506359 28781 leader_election.cc:288] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.506539 28614 consensus_queue.cc:260] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.507282 28614 ts_tablet_manager.cc:1398] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:12:02.507704 28614 tablet_bootstrap.cc:492] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:02.508693 28427 leader_election.cc:341] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.509868 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c875ec8c5e49d1bb20a23dd7a21902" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.510288 28715 raft_consensus.cc:482] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.510509 28715 raft_consensus.cc:504] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.511411 28715 leader_election.cc:288] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:02.512035 28614 tablet_bootstrap.cc:654] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:02.512655 28150 leader_election.cc:341] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.512846 28150 leader_election.cc:302] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:02.513341 28600 tablet_bootstrap.cc:492] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.513461 28604 tablet_bootstrap.cc:492] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.513572 28600 ts_tablet_manager.cc:1367] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.174s user 0.010s sys 0.014s
I20240503 14:12:02.513710 28604 ts_tablet_manager.cc:1367] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.005s
I20240503 14:12:02.514608 28600 raft_consensus.cc:348] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.514727 28604 raft_consensus.cc:348] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.514889 28600 raft_consensus.cc:374] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.514981 28600 raft_consensus.cc:729] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.514986 28604 raft_consensus.cc:374] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.515132 28604 raft_consensus.cc:729] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.515398 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559342d357d64f9eb9e3ab42ccb75942" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.515367 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559342d357d64f9eb9e3ab42ccb75942" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.515450 28604 consensus_queue.cc:260] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.515728 28216 raft_consensus.cc:2454] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.516068 28492 raft_consensus.cc:2454] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:02.516366 28604 ts_tablet_manager.cc:1398] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 14:12:02.516860 28600 consensus_queue.cc:260] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.517249 28290 leader_election.cc:302] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.519068 28604 tablet_bootstrap.cc:492] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.521325 28600 ts_tablet_manager.cc:1398] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:12:02.522358 28715 raft_consensus.cc:2781] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.522626 28715 raft_consensus.cc:482] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.522818 28715 raft_consensus.cc:3037] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.522945 28600 tablet_bootstrap.cc:492] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.530332 28600 tablet_bootstrap.cc:654] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.530331 28604 tablet_bootstrap.cc:654] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.531848 28603 tablet_replica.cc:330] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:02.532132 28603 raft_consensus.cc:2229] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:02.532466 28603 pending_rounds.cc:62] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:02.532549 28603 pending_rounds.cc:68] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597657103413248 op_type: NO_OP noop_request { }
I20240503 14:12:02.532728 28603 raft_consensus.cc:2866] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:02.532850 28603 raft_consensus.cc:2258] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.534225 28603 ts_tablet_manager.cc:1875] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.538559 28754 raft_consensus.cc:504] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.540292 28754 leader_election.cc:288] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.513341 28781 raft_consensus.cc:2726] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.545487 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb79edba14b646ef9be239b05a291d71" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:02.545838 28216 raft_consensus.cc:3037] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.555861 28614 tablet_bootstrap.cc:492] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.556139 28614 ts_tablet_manager.cc:1367] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.004s
I20240503 14:12:02.557214 28614 raft_consensus.cc:348] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.557524 28614 raft_consensus.cc:374] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.557623 28614 raft_consensus.cc:729] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.557332 28715 raft_consensus.cc:504] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.559064 28715 leader_election.cc:288] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:02.560247 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb79edba14b646ef9be239b05a291d71" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:02.560510 28357 raft_consensus.cc:3037] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.560612 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559342d357d64f9eb9e3ab42ccb75942" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:02.560878 28492 raft_consensus.cc:3037] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.563525 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559342d357d64f9eb9e3ab42ccb75942" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:02.563861 28217 raft_consensus.cc:3037] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.564121 28604 tablet_bootstrap.cc:492] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.564342 28604 ts_tablet_manager.cc:1367] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.003s
I20240503 14:12:02.565546 28604 raft_consensus.cc:348] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.565829 28604 raft_consensus.cc:374] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.565940 28604 raft_consensus.cc:729] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.566280 28604 consensus_queue.cc:260] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.568732 28604 ts_tablet_manager.cc:1398] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:02.569170 28604 tablet_bootstrap.cc:492] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.574585 28600 tablet_bootstrap.cc:492] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.574780 28600 ts_tablet_manager.cc:1367] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.007s
I20240503 14:12:02.575606 28600 raft_consensus.cc:348] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.575891 28600 raft_consensus.cc:374] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.576043 28600 raft_consensus.cc:729] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.576344 28600 consensus_queue.cc:260] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.572417 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6aa994dfa54a47408bfb34bcfda9a415 (table default.Table_4e83fe8e46644b8f8df871f6da4f0f1d [id=a1a30b8a5adb44b390e6bea2dd82c20c]) successfully deleted
I20240503 14:12:02.578645 28604 tablet_bootstrap.cc:654] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.580412 28600 ts_tablet_manager.cc:1398] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:02.581933 28600 tablet_bootstrap.cc:492] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.582337 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet eb79edba14b646ef9be239b05a291d71 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:02.582906 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet eee989ff083041dc8dcd2cd1b220780d
I20240503 14:12:02.583729 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_63619685889a4263ae065b254160cf30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.584482 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63619685889a4263ae065b254160cf30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.557904 28614 consensus_queue.cc:260] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.585426 28614 ts_tablet_manager.cc:1398] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240503 14:12:02.586126 28614 tablet_bootstrap.cc:492] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.586350 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.587031 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.588303 28614 tablet_bootstrap.cc:654] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.589442 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5eec9210201643c3a011af892345d421"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.590129 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5eec9210201643c3a011af892345d421 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.591109 28600 tablet_bootstrap.cc:654] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.597760 28604 tablet_bootstrap.cc:492] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.597795 28614 tablet_bootstrap.cc:492] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.597991 28604 ts_tablet_manager.cc:1367] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20240503 14:12:02.598021 28614 ts_tablet_manager.cc:1367] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240503 14:12:02.599010 28614 raft_consensus.cc:348] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.599328 28614 raft_consensus.cc:374] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.599231 28604 raft_consensus.cc:348] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.599465 28614 raft_consensus.cc:729] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.599557 28604 raft_consensus.cc:374] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.599682 28604 raft_consensus.cc:729] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.599735 28614 consensus_queue.cc:260] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.599958 28604 consensus_queue.cc:260] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.602272 28604 ts_tablet_manager.cc:1398] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:02.602722 28604 tablet_bootstrap.cc:492] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.605665 28604 tablet_bootstrap.cc:654] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.607151 28614 ts_tablet_manager.cc:1398] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20240503 14:12:02.607784 28614 tablet_bootstrap.cc:492] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.610265 28614 tablet_bootstrap.cc:654] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.613094 28600 tablet_bootstrap.cc:492] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.613337 28600 ts_tablet_manager.cc:1367] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.012s
I20240503 14:12:02.614466 28600 raft_consensus.cc:348] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.614789 28600 raft_consensus.cc:374] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.614897 28600 raft_consensus.cc:729] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.615229 28600 consensus_queue.cc:260] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.617410 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet eee989ff083041dc8dcd2cd1b220780d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.618253 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet eee989ff083041dc8dcd2cd1b220780d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.619073 28600 ts_tablet_manager.cc:1398] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 14:12:02.619725 28600 tablet_bootstrap.cc:492] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.622925 28614 tablet_bootstrap.cc:492] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.623183 28614 ts_tablet_manager.cc:1367] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240503 14:12:02.624189 28614 raft_consensus.cc:348] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.624495 28614 raft_consensus.cc:374] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.624606 28614 raft_consensus.cc:729] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.624943 28614 consensus_queue.cc:260] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.625659 28614 ts_tablet_manager.cc:1398] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.626145 28614 tablet_bootstrap.cc:492] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.627933 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet eee989ff083041dc8dcd2cd1b220780d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.630477 28614 tablet_bootstrap.cc:654] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.632107 28600 tablet_bootstrap.cc:654] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.631937 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7712b0c57d054073ac3f31fa183e72f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:02.632421 28604 tablet_bootstrap.cc:492] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.632652 28604 ts_tablet_manager.cc:1367] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.005s
W20240503 14:12:02.632812 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7712b0c57d054073ac3f31fa183e72f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.634743 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f2e8bc9e82a64d6682fa813e04fe7fc4" } new_table_name: "default.Table_2e135732daa240d7b537d589e0256892" modify_external_catalogs: true
W20240503 14:12:02.635980 28427 consensus_peers.cc:485] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87ae10db451b4e3a8a85d1d987c447d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:02.633661 28604 raft_consensus.cc:348] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.636354 28604 raft_consensus.cc:374] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.636454 28604 raft_consensus.cc:729] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.636828 28604 consensus_queue.cc:260] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.643174 28604 ts_tablet_manager.cc:1398] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:12:02.644546 28653 ts_tablet_manager.cc:1888] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.644800 28653 log.cc:1196] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/567f15be6b9e4baaba9587c3634f72ad
I20240503 14:12:02.645272 28653 ts_tablet_manager.cc:1909] T 567f15be6b9e4baaba9587c3634f72ad P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:02.648408 28494 raft_consensus.cc:2454] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.649005 28216 raft_consensus.cc:2454] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:02.649616 28357 raft_consensus.cc:2454] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:02.650326 28427 leader_election.cc:302] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.650522 28781 raft_consensus.cc:482] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.650786 28754 raft_consensus.cc:2781] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.650705 28781 raft_consensus.cc:504] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.650969 28754 raft_consensus.cc:686] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:02.651341 28754 consensus_queue.cc:237] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.651528 28781 leader_election.cc:288] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.654294 28290 leader_election.cc:302] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:02.661895 28604 tablet_bootstrap.cc:492] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.663655 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6186c8f2e546f39bf0f632d76166fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.664011 28357 raft_consensus.cc:2454] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:02.664817 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6186c8f2e546f39bf0f632d76166fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:02.665153 28494 raft_consensus.cc:2454] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:02.666996 28604 tablet_bootstrap.cc:654] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.667407 28150 leader_election.cc:302] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.678270 28614 tablet_bootstrap.cc:492] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.678256 28781 raft_consensus.cc:2781] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.678450 28781 raft_consensus.cc:482] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.678489 28614 ts_tablet_manager.cc:1367] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.012s
I20240503 14:12:02.678572 28781 raft_consensus.cc:3037] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.679549 28614 raft_consensus.cc:348] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.679826 28614 raft_consensus.cc:374] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.680069 28614 raft_consensus.cc:729] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.680388 28614 consensus_queue.cc:260] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.680794 28622 raft_consensus.cc:2781] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.681011 28622 raft_consensus.cc:686] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:02.681355 28622 consensus_queue.cc:237] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.682574 28614 ts_tablet_manager.cc:1398] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:02.683750 28614 tablet_bootstrap.cc:492] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.684806 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e145076fa64d4661bf830b556d2d2351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.685729 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e145076fa64d4661bf830b556d2d2351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.687214 28614 tablet_bootstrap.cc:654] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.692669 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7d209c20164a474485dae171a960b5f4" } new_table_name: "default.Table_666a0b3b2afb4a9fb91fb970e6c7a173" modify_external_catalogs: true
W20240503 14:12:02.706310 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 78987.4us
I20240503 14:12:02.707337 28614 tablet_bootstrap.cc:492] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.707588 28614 ts_tablet_manager.cc:1367] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20240503 14:12:02.707664 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 989ea2f962a7492d8c32e4becec1457d (DEFAULT_TABLE table=default.Table_da6c5739b62c43758e9e53182b790622 [id=6b342f8063bb4c2db3d8f63359afa1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.707893 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 989ea2f962a7492d8c32e4becec1457d (DEFAULT_TABLE table=default.Table_da6c5739b62c43758e9e53182b790622 [id=6b342f8063bb4c2db3d8f63359afa1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.708482 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 989ea2f962a7492d8c32e4becec1457d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.708650 28614 raft_consensus.cc:348] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.708915 28614 raft_consensus.cc:374] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.709013 28614 raft_consensus.cc:729] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.712869 28614 consensus_queue.cc:260] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.713660 28614 ts_tablet_manager.cc:1398] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:02.714146 28614 tablet_bootstrap.cc:492] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:02.716599 28614 tablet_bootstrap.cc:654] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.717418 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 59a91f22268346b38ae35f96b0d763f7 (DEFAULT_TABLE table=default.Table_59af1c61c0534502b0f2cdaff98f4e96 [id=434814ed785f43e380e9c8b1cd884cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.717427 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 989ea2f962a7492d8c32e4becec1457d (DEFAULT_TABLE table=default.Table_da6c5739b62c43758e9e53182b790622 [id=6b342f8063bb4c2db3d8f63359afa1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.717967 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a91f22268346b38ae35f96b0d763f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.676497 28600 tablet_bootstrap.cc:492] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.721443 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 59a91f22268346b38ae35f96b0d763f7 (DEFAULT_TABLE table=default.Table_59af1c61c0534502b0f2cdaff98f4e96 [id=434814ed785f43e380e9c8b1cd884cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.721891 28467 tablet_service.cc:1467] Processing CreateTablet for tablet f03bf91cf5414208b6767482223bc240 (DEFAULT_TABLE table=default.Table_7b0e6de295714677a8c47970ae8d75e8 [id=ddcf6334b4fc421f9abe196844fa69fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.722188 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5992e28f1745678618230c183609e4 (DEFAULT_TABLE table=default.Table_0fa4ad1ec7f348019705a71ba875350d [id=b1f8924b4e464e11aec7070b0e7c3951]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.721443 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (DEFAULT_TABLE table=default.Table_b47d551510f1443382dad930d5505634 [id=874fc82c70674371a8be82614bb6fcfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.722635 28464 tablet_service.cc:1467] Processing CreateTablet for tablet ffbd3291c56b4c13813c80d6775afe6f (DEFAULT_TABLE table=default.Table_f99793320ab94248abb70c15ee57ca02 [id=cde896fd6d7345f998085c10bbdee418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.729070 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 779f373ef12a4da091309fcf247b5693 (DEFAULT_TABLE table=default.Table_8543af24474940c3b1ed368e33cba893 [id=857b2f49969d4291b98de4f3776e627d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.729394 28334 tablet_service.cc:1467] Processing CreateTablet for tablet f51834695b8b4a729a088da81eee2e47 (DEFAULT_TABLE table=default.Table_c617bc5bcd1947f38f425f5624ec882c [id=743c045f2e104da8b09fd13245037eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.730038 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 351d5bfb12c34c5f9de90e2df1b36fc1 (DEFAULT_TABLE table=default.Table_2fbf7b7a2fb34274a2a21b61197964d7 [id=56f6e9c2bcac4a0db2e10e3d68e9f4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.729074 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5992e28f1745678618230c183609e4 (DEFAULT_TABLE table=default.Table_0fa4ad1ec7f348019705a71ba875350d [id=b1f8924b4e464e11aec7070b0e7c3951]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.731535 28492 raft_consensus.cc:2454] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.732198 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 989ea2f962a7492d8c32e4becec1457d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.732281 28603 ts_tablet_manager.cc:1888] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:02.732470 28603 log.cc:1196] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a50237f7cc324e61a27ef6fe2dd7f071
I20240503 14:12:02.733124 28603 ts_tablet_manager.cc:1909] T a50237f7cc324e61a27ef6fe2dd7f071 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:02.735186 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 59a91f22268346b38ae35f96b0d763f7 (DEFAULT_TABLE table=default.Table_59af1c61c0534502b0f2cdaff98f4e96 [id=434814ed785f43e380e9c8b1cd884cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.735942 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (DEFAULT_TABLE table=default.Table_b47d551510f1443382dad930d5505634 [id=874fc82c70674371a8be82614bb6fcfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.736575 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5992e28f1745678618230c183609e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.736647 28336 tablet_service.cc:1467] Processing CreateTablet for tablet ffbd3291c56b4c13813c80d6775afe6f (DEFAULT_TABLE table=default.Table_f99793320ab94248abb70c15ee57ca02 [id=cde896fd6d7345f998085c10bbdee418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.737284 28331 tablet_service.cc:1467] Processing CreateTablet for tablet f03bf91cf5414208b6767482223bc240 (DEFAULT_TABLE table=default.Table_7b0e6de295714677a8c47970ae8d75e8 [id=ddcf6334b4fc421f9abe196844fa69fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.737381 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51834695b8b4a729a088da81eee2e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.737972 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351d5bfb12c34c5f9de90e2df1b36fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.738780 28653 tablet_replica.cc:330] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:02.739589 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779f373ef12a4da091309fcf247b5693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.740928 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 351d5bfb12c34c5f9de90e2df1b36fc1 (DEFAULT_TABLE table=default.Table_2fbf7b7a2fb34274a2a21b61197964d7 [id=56f6e9c2bcac4a0db2e10e3d68e9f4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.740962 28471 tablet_service.cc:1467] Processing CreateTablet for tablet f51834695b8b4a729a088da81eee2e47 (DEFAULT_TABLE table=default.Table_c617bc5bcd1947f38f425f5624ec882c [id=743c045f2e104da8b09fd13245037eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.741434 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 779f373ef12a4da091309fcf247b5693 (DEFAULT_TABLE table=default.Table_8543af24474940c3b1ed368e33cba893 [id=857b2f49969d4291b98de4f3776e627d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.743122 28290 leader_election.cc:302] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.744019 28622 raft_consensus.cc:2781] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.744251 28622 raft_consensus.cc:686] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:02.744601 28622 consensus_queue.cc:237] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.745218 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5992e28f1745678618230c183609e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.745345 28661 ts_tablet_manager.cc:1888] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.746008 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1aa939aced4556ae0f0de0e62d1f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.746234 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03bf91cf5414208b6767482223bc240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.746460 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351d5bfb12c34c5f9de90e2df1b36fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.746687 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a91f22268346b38ae35f96b0d763f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.746002 28661 log.cc:1196] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/f35e87db1c2a42a79b426b2e7767367a
I20240503 14:12:02.746985 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779f373ef12a4da091309fcf247b5693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.747274 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 989ea2f962a7492d8c32e4becec1457d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.747861 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbd3291c56b4c13813c80d6775afe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.757297 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51834695b8b4a729a088da81eee2e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.758430 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a91f22268346b38ae35f96b0d763f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.759339 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03bf91cf5414208b6767482223bc240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.759940 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbd3291c56b4c13813c80d6775afe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.760646 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1aa939aced4556ae0f0de0e62d1f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.764756 28661 ts_tablet_manager.cc:1909] T f35e87db1c2a42a79b426b2e7767367a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:02.768595 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_068cea3b090241c7980d0edb57c05aec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.769351 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_068cea3b090241c7980d0edb57c05aec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.770967 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 59a91f22268346b38ae35f96b0d763f7
I20240503 14:12:02.771874 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2ad02256d653464e9528725b5e37c72c" } modify_external_catalogs: true
I20240503 14:12:02.772140 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2ad02256d653464e9528725b5e37c72c" } modify_external_catalogs: true
I20240503 14:12:02.773708 28604 tablet_bootstrap.cc:492] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.773784 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7fe5d280a1654d80a2176c1def459e2e
I20240503 14:12:02.773959 28604 ts_tablet_manager.cc:1367] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.112s user 0.012s sys 0.003s
I20240503 14:12:02.774382 28026 catalog_manager.cc:6554] ReplaceTablet: tablet 7fe5d280a1654d80a2176c1def459e2e deleted and replaced by tablet dda39f0323ec47799d3d44c96f3ba605
I20240503 14:12:02.774816 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2ef7a9ee753245dcad1ff57366d3eb99" } new_table_name: "default.Table_633e72d5fd384b689e09a7eef5a8d1d9" modify_external_catalogs: true
I20240503 14:12:02.774936 28604 raft_consensus.cc:348] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.775245 28604 raft_consensus.cc:374] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.775394 28604 raft_consensus.cc:729] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.775661 28604 consensus_queue.cc:260] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.776857 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_456be1a488a44191be1e21de223834c9 [id=548632b6d08e4435ad561b074e5e5b14]). Expected version 1 got 0
I20240503 14:12:02.777225 28033 catalog_manager.cc:5484] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.777755 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]). Expected version 3 got 0
I20240503 14:12:02.778301 28604 ts_tablet_manager.cc:1398] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:02.778672 28604 tablet_bootstrap.cc:492] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.779793 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_456be1a488a44191be1e21de223834c9" } new_table_name: "default.Table_17a1019962da48c0992b5452d4a67aae" modify_external_catalogs: true
I20240503 14:12:02.781311 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (DEFAULT_TABLE table=default.Table_b47d551510f1443382dad930d5505634 [id=874fc82c70674371a8be82614bb6fcfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.781877 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1aa939aced4556ae0f0de0e62d1f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.783946 28217 raft_consensus.cc:2454] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:02.784075 28600 ts_tablet_manager.cc:1367] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.100s user 0.001s sys 0.020s
I20240503 14:12:02.784968 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5992e28f1745678618230c183609e4 (DEFAULT_TABLE table=default.Table_0fa4ad1ec7f348019705a71ba875350d [id=b1f8924b4e464e11aec7070b0e7c3951]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.785472 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5992e28f1745678618230c183609e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.790356 28190 tablet_service.cc:1467] Processing CreateTablet for tablet f03bf91cf5414208b6767482223bc240 (DEFAULT_TABLE table=default.Table_7b0e6de295714677a8c47970ae8d75e8 [id=ddcf6334b4fc421f9abe196844fa69fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.791083 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03bf91cf5414208b6767482223bc240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.791235 28600 raft_consensus.cc:348] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.791488 28600 raft_consensus.cc:374] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.791631 28600 raft_consensus.cc:729] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.791915 28600 consensus_queue.cc:260] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.793344 28191 tablet_service.cc:1467] Processing CreateTablet for tablet f51834695b8b4a729a088da81eee2e47 (DEFAULT_TABLE table=default.Table_c617bc5bcd1947f38f425f5624ec882c [id=743c045f2e104da8b09fd13245037eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.793902 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 779f373ef12a4da091309fcf247b5693 (DEFAULT_TABLE table=default.Table_8543af24474940c3b1ed368e33cba893 [id=857b2f49969d4291b98de4f3776e627d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.794365 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 351d5bfb12c34c5f9de90e2df1b36fc1 (DEFAULT_TABLE table=default.Table_2fbf7b7a2fb34274a2a21b61197964d7 [id=56f6e9c2bcac4a0db2e10e3d68e9f4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.799950 28604 tablet_bootstrap.cc:654] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:02.804291 28427 consensus_peers.cc:485] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87ae10db451b4e3a8a85d1d987c447d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:02.805372 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51834695b8b4a729a088da81eee2e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.808357 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351d5bfb12c34c5f9de90e2df1b36fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.812315 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779f373ef12a4da091309fcf247b5693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.812716 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 4775ff1029e6449c83219030e3809984 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:02.815999 28192 tablet_service.cc:1467] Processing CreateTablet for tablet ffbd3291c56b4c13813c80d6775afe6f (DEFAULT_TABLE table=default.Table_f99793320ab94248abb70c15ee57ca02 [id=cde896fd6d7345f998085c10bbdee418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:02.822013 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbd3291c56b4c13813c80d6775afe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.823097 28600 ts_tablet_manager.cc:1398] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.038s user 0.000s sys 0.003s
I20240503 14:12:02.823546 28600 tablet_bootstrap.cc:492] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.826278 28781 raft_consensus.cc:504] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.827181 28781 leader_election.cc:288] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.827386 28653 raft_consensus.cc:2229] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:02.828217 28661 tablet_replica.cc:330] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:02.828374 28653 raft_consensus.cc:2258] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.828505 28661 raft_consensus.cc:2229] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:02.828917 28661 pending_rounds.cc:62] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:02.829042 28661 pending_rounds.cc:68] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597657326256128 op_type: NO_OP noop_request { }
I20240503 14:12:02.829209 28661 raft_consensus.cc:2866] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:02.829366 28661 raft_consensus.cc:2258] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:02.829972 28661 ts_tablet_manager.cc:1875] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:02.832401 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6186c8f2e546f39bf0f632d76166fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:02.832736 28492 raft_consensus.cc:3037] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:02.836256 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7fe5d280a1654d80a2176c1def459e2e (table default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7fe5d280a1654d80a2176c1def459e2e consensus state has no leader
I20240503 14:12:02.837594 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a91f22268346b38ae35f96b0d763f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.838148 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe5d280a1654d80a2176c1def459e2e with delete_type TABLET_DATA_DELETED (replaced by tablet dda39f0323ec47799d3d44c96f3ba605) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.838225 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 567f15be6b9e4baaba9587c3634f72ad (table default.Table_666d072a7d2a4affa01a3cf64abf777c [id=c094b16f94d44132bea89dc256965971]) successfully deleted
I20240503 14:12:02.838848 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a91f22268346b38ae35f96b0d763f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.839059 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet f35e87db1c2a42a79b426b2e7767367a (table default.Table_cc4fe0dac143484396ef862744cc9967 [id=e324426e1ff447a38c25b3867724fc4c]) successfully deleted
I20240503 14:12:02.839654 28653 ts_tablet_manager.cc:1875] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:02.841203 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fde0e59d911d4c1eb5125b1f869b396c consensus state has no leader
I20240503 14:12:02.842098 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 92c800e997e944158f7c839dd0e00d44 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:02.853978 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe5d280a1654d80a2176c1def459e2e with delete_type TABLET_DATA_DELETED (replaced by tablet dda39f0323ec47799d3d44c96f3ba605) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.861171 28603 tablet_replica.cc:330] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:02.861090 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6186c8f2e546f39bf0f632d76166fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:02.861079 28614 tablet_bootstrap.cc:492] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:02.862056 28614 ts_tablet_manager.cc:1367] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.148s user 0.016s sys 0.024s
I20240503 14:12:02.862092 28355 raft_consensus.cc:3037] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.864507 28754 raft_consensus.cc:482] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.864771 28754 raft_consensus.cc:504] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.865585 28754 leader_election.cc:288] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.867369 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacf2d1091ee458c961c691093ea65a9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.867700 28217 raft_consensus.cc:2454] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:02.868149 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a50237f7cc324e61a27ef6fe2dd7f071 (table default.Table_0515e83b3d914915a67f1d30f123eb09 [id=f699bab9cb6a4364a6b371df69e16222]) successfully deleted
W20240503 14:12:02.868577 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 71598d0a44384a3b93dec30d6c60ecd5 because the tablet was not found. No further retry: Not found: Tablet not found: 71598d0a44384a3b93dec30d6c60ecd5
I20240503 14:12:02.869503 28600 tablet_bootstrap.cc:654] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.871749 28427 leader_election.cc:302] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:02.872226 28754 raft_consensus.cc:2781] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.872375 28754 raft_consensus.cc:482] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.872514 28754 raft_consensus.cc:3037] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.878654 28634 raft_consensus.cc:482] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.878929 28634 raft_consensus.cc:504] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.879813 28634 leader_election.cc:288] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:02.881093 28427 leader_election.cc:341] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: e85b6db1a21c4bca80c8bdec17e45c59
I20240503 14:12:02.882366 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_71af49729f324730af4113012443a75c" } modify_external_catalogs: true
I20240503 14:12:02.882659 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_71af49729f324730af4113012443a75c" } modify_external_catalogs: true
I20240503 14:12:02.884300 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 4775ff1029e6449c83219030e3809984
I20240503 14:12:02.885107 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0fa4ad1ec7f348019705a71ba875350d" } new_table_name: "default.Table_fa5fb9616f9d41748b6c0b3e261e7f24" modify_external_catalogs: true
W20240503 14:12:02.885002 28736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:02.885493 28736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:02.885694 28736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:02.888146 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet ea3e9c030a424740bb2fb641328adfe1 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:02.890784 28028 catalog_manager.cc:5484] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.891428 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_456be1a488a44191be1e21de223834c9 [id=548632b6d08e4435ad561b074e5e5b14]). Expected version 1 got 0
I20240503 14:12:02.891830 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]). Expected version 3 got 0
I20240503 14:12:02.893236 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.894021 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:02.900271 28736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:02.900466 28736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:02.900563 28736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:02.900517 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b166fe6366ba496a869af1b106fe9d8e consensus state has no leader
W20240503 14:12:02.900692 28736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:02.901244 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cfc6b06d2df94f24867d0e7dd1be223d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.902055 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfc6b06d2df94f24867d0e7dd1be223d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.907166 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 4775ff1029e6449c83219030e3809984 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.907608 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 4775ff1029e6449c83219030e3809984 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.911715 28634 raft_consensus.cc:482] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.912067 28634 raft_consensus.cc:504] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.913096 28634 leader_election.cc:288] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.913355 28634 raft_consensus.cc:482] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.913538 28634 raft_consensus.cc:504] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.914409 28634 leader_election.cc:288] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:02.916010 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab05178dc081480ba409aab7f143f849" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:02.916378 28216 raft_consensus.cc:2454] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:02.917263 28427 leader_election.cc:302] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:02.917690 28427 leader_election.cc:341] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2502a13e8cbc4a3abfed225f94c44d2f
I20240503 14:12:02.917702 28812 raft_consensus.cc:2781] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.917873 28812 raft_consensus.cc:482] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.918037 28812 raft_consensus.cc:3037] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.920003 28736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.254:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.50.254
--webserver_port=38435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:02.920819 28692 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 14:12:02.922632 28736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:02.924182 28357 raft_consensus.cc:3037] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.924645 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacf2d1091ee458c961c691093ea65a9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:02.924114 28604 tablet_bootstrap.cc:492] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.925024 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe5d280a1654d80a2176c1def459e2e with delete_type TABLET_DATA_DELETED (replaced by tablet dda39f0323ec47799d3d44c96f3ba605) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.925426 28604 ts_tablet_manager.cc:1367] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.147s user 0.007s sys 0.008s
I20240503 14:12:02.926573 28604 raft_consensus.cc:348] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.926851 28604 raft_consensus.cc:374] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.926939 28604 raft_consensus.cc:729] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.926949 28736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:02.928054 28604 consensus_queue.cc:260] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.929260 28600 tablet_bootstrap.cc:492] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.932965 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cba5fd0a90c8433980631ba7e2048d8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:02.933790 28604 ts_tablet_manager.cc:1398] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
W20240503 14:12:02.933810 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba5fd0a90c8433980631ba7e2048d8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.934273 28604 tablet_bootstrap.cc:492] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.929502 28600 ts_tablet_manager.cc:1367] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.106s user 0.006s sys 0.032s
I20240503 14:12:02.935948 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ea3e9c030a424740bb2fb641328adfe1
I20240503 14:12:02.936643 28025 catalog_manager.cc:6554] ReplaceTablet: tablet ea3e9c030a424740bb2fb641328adfe1 deleted and replaced by tablet bd76060daab84c2f9f6727b8a9bb3a49
I20240503 14:12:02.937288 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_36c01c2a0d944899bf662092e1cc816d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.938153 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36c01c2a0d944899bf662092e1cc816d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.939872 28492 raft_consensus.cc:2454] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:02.941020 28653 ts_tablet_manager.cc:1888] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:02.941016 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6d7feb39cce84e66b75914a6bcbcfcc9" } modify_external_catalogs: true
I20240503 14:12:02.941211 28653 log.cc:1196] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ec6186c8f2e546f39bf0f632d76166fa
I20240503 14:12:02.941290 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6d7feb39cce84e66b75914a6bcbcfcc9" } modify_external_catalogs: true
I20240503 14:12:02.941670 28653 ts_tablet_manager.cc:1909] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:02.942924 28600 raft_consensus.cc:348] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.943208 28600 raft_consensus.cc:374] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.943305 28600 raft_consensus.cc:729] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.945101 28604 tablet_bootstrap.cc:654] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.945827 28150 leader_election.cc:302] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:02.946274 28150 raft_consensus.cc:2670] T ec6186c8f2e546f39bf0f632d76166fa P a64bc15d65d84ad5a005960a9a84df9c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:02.946667 28600 consensus_queue.cc:260] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.949936 28600 ts_tablet_manager.cc:1398] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240503 14:12:02.951514 28600 tablet_bootstrap.cc:492] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.951287 28614 raft_consensus.cc:348] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.951635 28614 raft_consensus.cc:374] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.951763 28614 raft_consensus.cc:729] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.952041 28614 consensus_queue.cc:260] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.952833 28614 ts_tablet_manager.cc:1398] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.091s user 0.002s sys 0.000s
I20240503 14:12:02.953281 28614 tablet_bootstrap.cc:492] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:02.953416 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9609112657f427cbeeb3ef8f28cddf0 has no consensus state
I20240503 14:12:02.954089 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.954838 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.955462 28600 tablet_bootstrap.cc:654] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.956313 28030 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet dd4c34703a0b46c4a6cf756326602080
I20240503 14:12:02.956849 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_20365e1d61b74cb8b39b9a51c2d608c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.957629 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20365e1d61b74cb8b39b9a51c2d608c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.960009 28604 tablet_bootstrap.cc:492] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:02.960063 28356 raft_consensus.cc:2454] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:02.960283 28604 ts_tablet_manager.cc:1367] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20240503 14:12:02.961319 28604 raft_consensus.cc:348] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.961592 28604 raft_consensus.cc:374] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.962194 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_037096975b83431e99d595ed4e62d706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:02.962877 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_037096975b83431e99d595ed4e62d706 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:02.963342 28604 raft_consensus.cc:729] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.963762 28604 consensus_queue.cc:260] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.965817 28711 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 14:12:02.967639 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet bacf2d1091ee458c961c691093ea65a9
I20240503 14:12:02.973614 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3e9c030a424740bb2fb641328adfe1 with delete_type TABLET_DATA_DELETED (replaced by tablet bd76060daab84c2f9f6727b8a9bb3a49) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.975649 28217 raft_consensus.cc:1261] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:02.975847 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet bacf2d1091ee458c961c691093ea65a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.978827 28600 tablet_bootstrap.cc:492] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.979049 28600 ts_tablet_manager.cc:1367] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.011s
I20240503 14:12:02.980147 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet ab05178dc081480ba409aab7f143f849 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.980206 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet bacf2d1091ee458c961c691093ea65a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:02.980526 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3e9c030a424740bb2fb641328adfe1 with delete_type TABLET_DATA_DELETED (replaced by tablet bd76060daab84c2f9f6727b8a9bb3a49) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.980736 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4775ff1029e6449c83219030e3809984 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.980818 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 6106d52df3c54c2bb4abadd8605cd912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:00 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:02.980805 28600 raft_consensus.cc:348] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:02.981016 28600 raft_consensus.cc:374] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.981146 28600 raft_consensus.cc:729] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.981438 28600 consensus_queue.cc:260] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:02.981753 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a91f22268346b38ae35f96b0d763f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:02.981000 28604 ts_tablet_manager.cc:1398] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:12:02.982573 28604 tablet_bootstrap.cc:492] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:02.982578 28600 ts_tablet_manager.cc:1398] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:02.982905 28600 tablet_bootstrap.cc:492] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:02.985399 28600 tablet_bootstrap.cc:654] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.986166 28604 tablet_bootstrap.cc:654] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.986272 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_63619685889a4263ae065b254160cf30" } modify_external_catalogs: true
I20240503 14:12:02.986513 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_63619685889a4263ae065b254160cf30" } modify_external_catalogs: true
I20240503 14:12:02.987382 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_45bc934b7dba4fbc966160c359918f67" } new_table_name: "default.Table_b052ceeaac3644e18e8f3cb1947a3863" modify_external_catalogs: true
I20240503 14:12:02.990402 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3e9c030a424740bb2fb641328adfe1 with delete_type TABLET_DATA_DELETED (replaced by tablet bd76060daab84c2f9f6727b8a9bb3a49) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:02.992925 28427 leader_election.cc:341] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: e85b6db1a21c4bca80c8bdec17e45c59
I20240503 14:12:02.993119 28427 leader_election.cc:302] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:02.994127 28634 raft_consensus.cc:2726] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.994405 28600 tablet_bootstrap.cc:492] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:02.994652 28600 ts_tablet_manager.cc:1367] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:12:02.995007 28634 consensus_queue.cc:1035] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.995748 28600 raft_consensus.cc:348] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.996048 28600 raft_consensus.cc:374] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.996244 28600 raft_consensus.cc:729] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
W20240503 14:12:02.996515 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b5992e28f1745678618230c183609e4 consensus state has no leader
I20240503 14:12:02.996539 28600 consensus_queue.cc:260] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:02.997680 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9dbabe5beb9a42c69cb64e1b364d75a1
I20240503 14:12:02.998456 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet e85b6db1a21c4bca80c8bdec17e45c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.005107 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.005681 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:03.006378 28600 ts_tablet_manager.cc:1398] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240503 14:12:03.008421 28600 tablet_bootstrap.cc:492] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:03.013989 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_419d8eaa44c34cca86e012471fd9814d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.014672 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f with delete_type TABLET_DATA_DELETED (replaced by tablet da19787034c24ebe929601d49cb21b3b) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:03.014801 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_419d8eaa44c34cca86e012471fd9814d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.015348 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet bacf2d1091ee458c961c691093ea65a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:03.017346 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:03.019716 28820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:03.021425 28818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:03.024443 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c800e997e944158f7c839dd0e00d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:03.023999 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2502a13e8cbc4a3abfed225f94c44d2f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.025130 28352 raft_consensus.cc:2454] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:03.026705 28661 ts_tablet_manager.cc:1888] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:03.026901 28661 log.cc:1196] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/87ae10db451b4e3a8a85d1d987c447d9
I20240503 14:12:03.027431 28661 ts_tablet_manager.cc:1909] T 87ae10db451b4e3a8a85d1d987c447d9 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:03.028987 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]). Expected version 2 got 0
I20240503 14:12:03.029346 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 2ef6cd66f34c4980bad5e500309b1568 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]). Expected version 1 got 0
I20240503 14:12:03.029760 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f17acf95467b4c69ad200cf775572a31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.030565 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f17acf95467b4c69ad200cf775572a31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.031288 28754 raft_consensus.cc:504] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.032316 28754 leader_election.cc:288] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.035367 28604 tablet_bootstrap.cc:492] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:03.035594 28604 ts_tablet_manager.cc:1367] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.003s
I20240503 14:12:03.036260 28653 tablet_replica.cc:330] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:03.036546 28653 raft_consensus.cc:2229] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:03.036804 28653 raft_consensus.cc:2258] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:03.036803 28604 raft_consensus.cc:348] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:03.039216 28192 tablet_service.cc:1467] Processing CreateTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b (DEFAULT_TABLE table=default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f [id=837e8a35f3024ddcbe19763625d82a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.039698 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae4969fc9474fdbaeae1b4d09e4875b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.040877 28197 tablet_service.cc:1467] Processing CreateTablet for tablet cc230b489ab8455c8cb2eba31084fddb (DEFAULT_TABLE table=default.Table_c2d18c6af36b4d0096425aa0afc52fd2 [id=084b186eba154ce1939f530379f4ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.041385 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc230b489ab8455c8cb2eba31084fddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.043442 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 6dab0296a0894a63afc16ed466c591d5 (DEFAULT_TABLE table=default.Table_de96804c306a4b0e91a2e6bffd5943a8 [id=1b1ed3d3a0b24ced96ba66459ccecc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.043951 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dab0296a0894a63afc16ed466c591d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.044548 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e (DEFAULT_TABLE table=default.Table_986d778bcd814c9c87dbd18d1f485cc9 [id=5963eb580afa4dc18cf4e472ed1fb76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.044994 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75dafd98b05d4c63bbf30165a2ac8d1e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:03.045827 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f45eb624434a404d85dcd233721ebf68 consensus state has no leader
I20240503 14:12:03.049394 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 40da5f5fff94462d8af740f3ee337e76 (DEFAULT_TABLE table=default.Table_63619685889a4263ae065b254160cf30 [id=6a81b734738e4d998215938d5ca5dad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.050144 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40da5f5fff94462d8af740f3ee337e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.025493 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab05178dc081480ba409aab7f143f849" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:03.050161 28824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:03.050874 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacf2d1091ee458c961c691093ea65a9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:03.051218 28217 raft_consensus.cc:3037] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.052867 28736 server_base.cc:1034] running on GCE node
I20240503 14:12:03.052919 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b47d551510f1443382dad930d5505634" } new_table_name: "default.Table_3f8c7245a7a84b1e904cf130d5089dda" modify_external_catalogs: true
I20240503 14:12:03.054304 28736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:03.054484 28604 raft_consensus.cc:374] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.054726 28604 raft_consensus.cc:729] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.054533 28427 leader_election.cc:302] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:03.056151 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_17a1019962da48c0992b5452d4a67aae [id=548632b6d08e4435ad561b074e5e5b14]). Expected version 2 got 0
I20240503 14:12:03.056314 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e (DEFAULT_TABLE table=default.Table_986d778bcd814c9c87dbd18d1f485cc9 [id=5963eb580afa4dc18cf4e472ed1fb76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.056511 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 40da5f5fff94462d8af740f3ee337e76 (DEFAULT_TABLE table=default.Table_63619685889a4263ae065b254160cf30 [id=6a81b734738e4d998215938d5ca5dad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.056617 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 7de7832122804eada175f710cc5bab5e (table default.Table_cc4fe0dac143484396ef862744cc9967 [id=e324426e1ff447a38c25b3867724fc4c]). Expected version 1 got 0
I20240503 14:12:03.057642 28610 raft_consensus.cc:482] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.057889 28754 raft_consensus.cc:2781] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.057830 28610 raft_consensus.cc:504] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.058696 28610 leader_election.cc:288] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.058818 28464 tablet_service.cc:1467] Processing CreateTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b (DEFAULT_TABLE table=default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f [id=837e8a35f3024ddcbe19763625d82a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.059165 28653 ts_tablet_manager.cc:1875] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:03.059365 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 41cbd1326869434594e126421ad132d8 (DEFAULT_TABLE table=default.Table_a07e2f0650084270860bd28e40f224aa [id=a98b2a6e32bf425eba5d2e039def6b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.059545 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef6cd66f34c4980bad5e500309b1568" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.059710 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 6dab0296a0894a63afc16ed466c591d5 (DEFAULT_TABLE table=default.Table_de96804c306a4b0e91a2e6bffd5943a8 [id=1b1ed3d3a0b24ced96ba66459ccecc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.060021 28474 tablet_service.cc:1467] Processing CreateTablet for tablet cc230b489ab8455c8cb2eba31084fddb (DEFAULT_TABLE table=default.Table_c2d18c6af36b4d0096425aa0afc52fd2 [id=084b186eba154ce1939f530379f4ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.060427 28754 raft_consensus.cc:482] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.060577 28754 raft_consensus.cc:3037] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.060863 28604 consensus_queue.cc:260] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:03.061349 28150 leader_election.cc:341] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.067154 28600 tablet_bootstrap.cc:654] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.069025 28692 tablet.cc:1722] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:03.072273 28191 tablet_service.cc:1467] Processing CreateTablet for tablet e9609112657f427cbeeb3ef8f28cddf0 (DEFAULT_TABLE table=default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.072394 28604 ts_tablet_manager.cc:1398] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.037s user 0.003s sys 0.004s
I20240503 14:12:03.072772 28604 tablet_bootstrap.cc:492] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:03.072798 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9609112657f427cbeeb3ef8f28cddf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.074345 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 41cbd1326869434594e126421ad132d8 (DEFAULT_TABLE table=default.Table_a07e2f0650084270860bd28e40f224aa [id=a98b2a6e32bf425eba5d2e039def6b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.075062 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cbd1326869434594e126421ad132d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.077883 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 41cbd1326869434594e126421ad132d8 (DEFAULT_TABLE table=default.Table_a07e2f0650084270860bd28e40f224aa [id=a98b2a6e32bf425eba5d2e039def6b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.079020 28604 tablet_bootstrap.cc:654] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.080878 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 6dab0296a0894a63afc16ed466c591d5 (DEFAULT_TABLE table=default.Table_de96804c306a4b0e91a2e6bffd5943a8 [id=1b1ed3d3a0b24ced96ba66459ccecc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.081380 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dab0296a0894a63afc16ed466c591d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.082506 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 40da5f5fff94462d8af740f3ee337e76 (DEFAULT_TABLE table=default.Table_63619685889a4263ae065b254160cf30 [id=6a81b734738e4d998215938d5ca5dad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.082993 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40da5f5fff94462d8af740f3ee337e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.084498 28715 consensus_queue.cc:1035] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.085332 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.086141 28736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:03.087714 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cbd1326869434594e126421ad132d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.088757 28194 tablet_service.cc:1467] Processing CreateTablet for tablet a32b6995818041778617a0d6bf936c8c (DEFAULT_TABLE table=default.Table_5a5d61e70c514fbfac570c0d594e7010 [id=e2f2ebf7731a41e3ae451eddf555c454]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.089532 28351 raft_consensus.cc:2454] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:03.090322 28187 tablet_service.cc:1467] Processing CreateTablet for tablet f50ad403bd354559b2522a76af1fa722 (DEFAULT_TABLE table=default.Table_5eec9210201643c3a011af892345d421 [id=f4c76f2fa96d41c491fb0ac30f49e43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.091279 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef6cd66f34c4980bad5e500309b1568" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.091723 28353 raft_consensus.cc:2454] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:03.091828 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_afe079d9fefc4dd3af481024b84ce6a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.092558 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afe079d9fefc4dd3af481024b84ce6a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.080397 28327 tablet_service.cc:1467] Processing CreateTablet for tablet e9609112657f427cbeeb3ef8f28cddf0 (DEFAULT_TABLE table=default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.094770 28736 hybrid_clock.cc:648] HybridClock initialized: now 1714745523094627 us; error 49 us; skew 500 ppm
I20240503 14:12:03.095933 28736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:03.099027 28466 tablet_service.cc:1467] Processing CreateTablet for tablet a32b6995818041778617a0d6bf936c8c (DEFAULT_TABLE table=default.Table_5a5d61e70c514fbfac570c0d594e7010 [id=e2f2ebf7731a41e3ae451eddf555c454]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.084115 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.099561 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.099692 28150 leader_election.cc:302] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:03.099898 28473 tablet_service.cc:1467] Processing CreateTablet for tablet f50ad403bd354559b2522a76af1fa722 (DEFAULT_TABLE table=default.Table_5eec9210201643c3a011af892345d421 [id=f4c76f2fa96d41c491fb0ac30f49e43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.100250 28469 tablet_service.cc:1467] Processing CreateTablet for tablet e9609112657f427cbeeb3ef8f28cddf0 (DEFAULT_TABLE table=default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.100455 28610 raft_consensus.cc:2781] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.100538 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cfc6b06d2df94f24867d0e7dd1be223d" } modify_external_catalogs: true
I20240503 14:12:03.100663 28610 raft_consensus.cc:482] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.100623 28829 tablet.cc:1722] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:03.100785 28610 raft_consensus.cc:3037] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.100842 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cfc6b06d2df94f24867d0e7dd1be223d" } modify_external_catalogs: true
I20240503 14:12:03.099373 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bacf2d1091ee458c961c691093ea65a9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:03.102613 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e (DEFAULT_TABLE table=default.Table_986d778bcd814c9c87dbd18d1f485cc9 [id=5963eb580afa4dc18cf4e472ed1fb76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.107317 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b85675e32b564761a3df4d8b9a1cbc45" } new_table_name: "default.Table_cef464f155254da3a6e7a33965f98e2d" modify_external_catalogs: true
I20240503 14:12:03.111519 28352 raft_consensus.cc:3037] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.118039 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9609112657f427cbeeb3ef8f28cddf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.115324 28330 tablet_service.cc:1467] Processing CreateTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b (DEFAULT_TABLE table=default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f [id=837e8a35f3024ddcbe19763625d82a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.121526 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 40da5f5fff94462d8af740f3ee337e76
I20240503 14:12:03.124717 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 40da5f5fff94462d8af740f3ee337e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:03.124827 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f1aa939aced4556ae0f0de0e62d1f97 consensus state has no leader
I20240503 14:12:03.127894 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50ad403bd354559b2522a76af1fa722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.128211 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74df7781cc6a4dd6b6fd78a728ffd71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.130798 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32b6995818041778617a0d6bf936c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.133610 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75dafd98b05d4c63bbf30165a2ac8d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.134774 28736 webserver.cc:460] Webserver started at http://127.27.50.254:38435/ using document root <none> and password file <none>
I20240503 14:12:03.135071 28781 raft_consensus.cc:482] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.135325 28781 raft_consensus.cc:504] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.136265 28781 leader_election.cc:288] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.138427 28736 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:03.138895 28614 tablet_bootstrap.cc:654] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.139854 28326 tablet_service.cc:1467] Processing CreateTablet for tablet a32b6995818041778617a0d6bf936c8c (DEFAULT_TABLE table=default.Table_5a5d61e70c514fbfac570c0d594e7010 [id=e2f2ebf7731a41e3ae451eddf555c454]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.140913 28736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:03.141598 28325 tablet_service.cc:1467] Processing CreateTablet for tablet f50ad403bd354559b2522a76af1fa722 (DEFAULT_TABLE table=default.Table_5eec9210201643c3a011af892345d421 [id=f4c76f2fa96d41c491fb0ac30f49e43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.143644 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 40da5f5fff94462d8af740f3ee337e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:03.145658 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5ddacd70c694443e8f6e5639e1ea2afc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.146667 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ddacd70c694443e8f6e5639e1ea2afc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.147794 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23186a7a0dfb432e813c8a05e97e46dd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.148901 28715 consensus_queue.cc:1035] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240503 14:12:03.151543 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c9442a77363b44c584467e25ecc9878f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.152508 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9442a77363b44c584467e25ecc9878f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.160313 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74df7781cc6a4dd6b6fd78a728ffd71b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:03.161760 28150 leader_election.cc:341] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.165086 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 40da5f5fff94462d8af740f3ee337e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.167078 28332 tablet_service.cc:1467] Processing CreateTablet for tablet cc230b489ab8455c8cb2eba31084fddb (DEFAULT_TABLE table=default.Table_c2d18c6af36b4d0096425aa0afc52fd2 [id=084b186eba154ce1939f530379f4ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.169116 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae4969fc9474fdbaeae1b4d09e4875b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.169660 28715 raft_consensus.cc:482] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.171844 28715 raft_consensus.cc:504] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.172346 28027 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_2e135732daa240d7b537d589e0256892 [id=3a24797cc53b439da44cc77025fa9f92] alter complete (version 5)
I20240503 14:12:03.173628 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c800e997e944158f7c839dd0e00d44" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.174079 28492 raft_consensus.cc:2454] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:03.174700 28290 leader_election.cc:302] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:03.175066 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23186a7a0dfb432e813c8a05e97e46dd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.181057 28715 leader_election.cc:288] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:03.181366 28715 raft_consensus.cc:2781] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.181586 28715 raft_consensus.cc:482] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.181756 28715 raft_consensus.cc:3037] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:03.181717 28150 leader_election.cc:341] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.181898 28150 leader_election.cc:302] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:03.184805 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_22032e3385624e98a1b0bb1484f51c64" } modify_external_catalogs: true
I20240503 14:12:03.185197 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_22032e3385624e98a1b0bb1484f51c64" } modify_external_catalogs: true
I20240503 14:12:03.185909 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f" } modify_external_catalogs: true
I20240503 14:12:03.186129 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f" } modify_external_catalogs: true
I20240503 14:12:03.192365 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9d94370dfab6449ea9ad79c5530f078a" } new_table_name: "default.Table_085095beb9c0496385c83ff148946b68" modify_external_catalogs: true
I20240503 14:12:03.200354 28024 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b55bb985dadc41caa2e01939c2f04805
I20240503 14:12:03.200469 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]). Expected version 3 got 0
W20240503 14:12:03.202205 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd76060daab84c2f9f6727b8a9bb3a49 has no consensus state
I20240503 14:12:03.209455 28836 raft_consensus.cc:2726] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:03.216573 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_652b2cc6b6024e738ecdd3cb52306637"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.217482 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_652b2cc6b6024e738ecdd3cb52306637 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:03.219233 28819 log.cc:924] Time spent T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20240503 14:12:03.221413 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3a2892577d0744b5927abb18464fb658" } modify_external_catalogs: true
I20240503 14:12:03.221760 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3a2892577d0744b5927abb18464fb658" } modify_external_catalogs: true
I20240503 14:12:03.224058 28355 raft_consensus.cc:2454] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:03.224385 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.225132 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.225522 28812 raft_consensus.cc:504] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.226497 28812 leader_election.cc:288] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.226527 28603 raft_consensus.cc:2229] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:03.226850 28831 log.cc:924] Time spent T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240503 14:12:03.226971 28603 raft_consensus.cc:2258] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:03.227185 28217 raft_consensus.cc:2454] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:03.227741 28603 ts_tablet_manager.cc:1875] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:03.228844 28754 raft_consensus.cc:504] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.229064 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 75) took 1319 ms. Trace:
I20240503 14:12:03.229617 28754 leader_election.cc:288] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.229298 28661 rpcz_store.cc:276] 0503 14:12:01.909327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:01.909376 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:02.828125 (+918749us) ts_tablet_manager.cc:1120] Deleting tablet 87ae10db451b4e3a8a85d1d987c447d9
0503 14:12:02.828137 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:02.930574 (+102437us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.026491 (+ 95917us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.228981 (+202490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":918483}
I20240503 14:12:03.230695 28661 tablet_replica.cc:330] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:03.230981 28661 raft_consensus.cc:2229] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:03.231192 28661 raft_consensus.cc:2258] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:03.233494 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75dafd98b05d4c63bbf30165a2ac8d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.234279 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40da5f5fff94462d8af740f3ee337e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.234867 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae4969fc9474fdbaeae1b4d09e4875b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.235423 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cbd1326869434594e126421ad132d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.235973 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dab0296a0894a63afc16ed466c591d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.236280 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc230b489ab8455c8cb2eba31084fddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.236608 28427 leader_election.cc:302] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:03.236797 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32b6995818041778617a0d6bf936c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.237002 28754 raft_consensus.cc:2781] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.237305 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74df7781cc6a4dd6b6fd78a728ffd71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.237830 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50ad403bd354559b2522a76af1fa722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.238283 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9609112657f427cbeeb3ef8f28cddf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.245103 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab05178dc081480ba409aab7f143f849" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:03.245483 28217 raft_consensus.cc:3037] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.248824 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_dbb71b03f52c48d29206e945b956f702"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.251294 28836 raft_consensus.cc:482] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.251545 28836 raft_consensus.cc:504] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.252514 28836 leader_election.cc:288] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.253499 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1de9717d0e44179b875174d5a76e4c3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.253824 28494 raft_consensus.cc:2454] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:03.254508 28150 leader_election.cc:302] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:03.254917 28836 raft_consensus.cc:2781] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.255045 28836 raft_consensus.cc:482] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.255180 28836 raft_consensus.cc:3037] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:03.255617 28837 log.cc:924] Time spent T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240503 14:12:03.263743 28610 raft_consensus.cc:504] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:03.265173 28427 leader_election.cc:341] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2502a13e8cbc4a3abfed225f94c44d2f
I20240503 14:12:03.266362 28604 tablet_bootstrap.cc:492] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:03.266598 28604 ts_tablet_manager.cc:1367] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.194s user 0.012s sys 0.028s
I20240503 14:12:03.267786 28604 raft_consensus.cc:348] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.268039 28604 raft_consensus.cc:374] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.268200 28604 raft_consensus.cc:729] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.268514 28604 consensus_queue.cc:260] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.269613 28736 fs_manager.cc:709] Time spent opening directory manager: real 0.122s user 0.008s sys 0.001s
I20240503 14:12:03.270277 28604 ts_tablet_manager.cc:1398] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:03.271031 28604 tablet_bootstrap.cc:492] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:03.276110 28604 tablet_bootstrap.cc:654] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.278752 28812 raft_consensus.cc:482] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.278990 28812 raft_consensus.cc:504] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.279760 28812 leader_election.cc:288] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.296279 28846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:03.301812 28736 fs_manager.cc:725] Time spent opening block manager: real 0.028s user 0.004s sys 0.006s
I20240503 14:12:03.302800 28622 raft_consensus.cc:482] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.303081 28622 raft_consensus.cc:504] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.304299 28622 leader_election.cc:288] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:03.304350 28604 tablet_bootstrap.cc:492] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:03.304687 28604 ts_tablet_manager.cc:1367] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.002s
I20240503 14:12:03.223902 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9a954e776a184d819708a62a99ed94fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.307094 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32b6995818041778617a0d6bf936c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.306660 28604 raft_consensus.cc:348] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.307827 28604 raft_consensus.cc:374] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.308347 28604 raft_consensus.cc:729] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.308863 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50ad403bd354559b2522a76af1fa722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.310200 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc230b489ab8455c8cb2eba31084fddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.310238 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1de9717d0e44179b875174d5a76e4c3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.311478 28604 consensus_queue.cc:260] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:03.249794 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbb71b03f52c48d29206e945b956f702 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:03.307058 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a954e776a184d819708a62a99ed94fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.317569 28843 tablet.cc:1722] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 14:12:03.318352 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c617bc5bcd1947f38f425f5624ec882c" } new_table_name: "default.Table_b0e228f901e4427ca9bbfc14b02c1f7f" modify_external_catalogs: true
I20240503 14:12:03.302277 28736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
uuid: "bb6a7aee99214df4b02858ddae025b14"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-8wrt"
I20240503 14:12:03.266705 28600 tablet_bootstrap.cc:492] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:03.335475 28600 ts_tablet_manager.cc:1367] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.327s user 0.023s sys 0.024s
I20240503 14:12:03.336709 28600 raft_consensus.cc:348] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.336974 28600 raft_consensus.cc:374] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.337106 28600 raft_consensus.cc:729] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.337379 28600 consensus_queue.cc:260] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.338106 28600 ts_tablet_manager.cc:1398] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:03.338524 28600 tablet_bootstrap.cc:492] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:03.237318 28754 raft_consensus.cc:686] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:03.280530 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23186a7a0dfb432e813c8a05e97e46dd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:03.340270 28216 raft_consensus.cc:2454] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:03.312615 28150 leader_election.cc:341] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.342692 28025 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_17a1019962da48c0992b5452d4a67aae [id=548632b6d08e4435ad561b074e5e5b14] alter complete (version 2)
I20240503 14:12:03.342734 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet cae4969fc9474fdbaeae1b4d09e4875b
I20240503 14:12:03.343840 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef6cd66f34c4980bad5e500309b1568" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:03.344157 28354 raft_consensus.cc:3037] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:03.344460 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13c875ec8c5e49d1bb20a23dd7a21902 consensus state has no leader
I20240503 14:12:03.344615 28781 raft_consensus.cc:482] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.344817 28781 raft_consensus.cc:504] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.345674 28781 leader_election.cc:288] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.347205 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45eb624434a404d85dcd233721ebf68" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.347327 28736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:03.348507 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:03.338541 28610 leader_election.cc:288] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:03.349630 28150 leader_election.cc:341] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.349961 28604 ts_tablet_manager.cc:1398] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.042s user 0.003s sys 0.000s
I20240503 14:12:03.350239 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef6cd66f34c4980bad5e500309b1568" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:03.350780 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45eb624434a404d85dcd233721ebf68" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.351341 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.351770 28427 leader_election.cc:302] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:03.353767 28150 leader_election.cc:341] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.358304 28600 tablet_bootstrap.cc:654] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.368674 28653 ts_tablet_manager.cc:1888] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:12:03.369122 28653 log.cc:1196] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/5d637087c60144aaaa9b0965110d280e
I20240503 14:12:03.369244 28352 raft_consensus.cc:2454] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:03.369786 28653 ts_tablet_manager.cc:1909] T 5d637087c60144aaaa9b0965110d280e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:03.371361 28661 ts_tablet_manager.cc:1875] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:03.371858 28811 log.cc:924] Time spent T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Append to log took a long time: real 0.116s user 0.002s sys 0.000s
I20240503 14:12:03.372169 28754 consensus_queue.cc:237] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.376893 28836 raft_consensus.cc:504] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.377866 28836 leader_election.cc:288] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.379014 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1de9717d0e44179b875174d5a76e4c3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:03.379860 28150 leader_election.cc:341] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.384456 28604 tablet_bootstrap.cc:492] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:03.397116 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e3e1acbcbb7c4aeab3db92cc253b441d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.406913 28600 tablet_bootstrap.cc:492] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:03.407239 28600 ts_tablet_manager.cc:1367] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.004s
I20240503 14:12:03.408571 28600 raft_consensus.cc:348] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:03.408879 28600 raft_consensus.cc:374] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.409003 28600 raft_consensus.cc:729] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.409487 28600 consensus_queue.cc:260] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:03.410343 28600 ts_tablet_manager.cc:1398] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:03.410817 28600 tablet_bootstrap.cc:492] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:03.413694 28600 tablet_bootstrap.cc:654] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:03.425824 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3e1acbcbb7c4aeab3db92cc253b441d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.398269 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f17acf95467b4c69ad200cf775572a31" } new_table_name: "default.Table_fc0d4a5f37ae46d9807079682b94ba1d" modify_external_catalogs: true
I20240503 14:12:03.426893 28185 tablet_service.cc:1467] Processing CreateTablet for tablet dda39f0323ec47799d3d44c96f3ba605 (DEFAULT_TABLE table=default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.428148 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 63e977f7837e49fb93ef0fd18b619792 (DEFAULT_TABLE table=default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4 [id=dcdb4c8fe4774779b0303db262dbfcc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.428198 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 3514b79467dc42bc89fdacf8e39a000d (DEFAULT_TABLE table=default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.426970 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 (DEFAULT_TABLE table=default.Table_068cea3b090241c7980d0edb57c05aec [id=80b934d1638444ebb855ebacf917c23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.431620 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]). Expected version 2 got 0
I20240503 14:12:03.417395 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5264956c580441a6a23c96d39a41ea71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.418401 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc4fe0dac143484396ef862744cc9967" } modify_external_catalogs: true
I20240503 14:12:03.436362 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda39f0323ec47799d3d44c96f3ba605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.436973 28191 tablet_service.cc:1467] Processing CreateTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 (DEFAULT_TABLE table=default.Table_cef464f155254da3a6e7a33965f98e2d [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.449893 28836 raft_consensus.cc:482] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.460213 28354 raft_consensus.cc:2454] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:03.475757 28150 leader_election.cc:302] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:03.476683 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 7945118bb17b431dac8670007d2d0a18 (DEFAULT_TABLE table=default.Table_e145076fa64d4661bf830b556d2d2351 [id=892862b894c547e384ebf8b33b9a7ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.473448 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 93462d2b1a474bd688d6266cd3b0695c (DEFAULT_TABLE table=default.Table_7712b0c57d054073ac3f31fa183e72f1 [id=48c3f3e039044d10a6cfa1b6bcd5ae79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.477332 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 129717ef137c41c5a08d401c64164187 (DEFAULT_TABLE table=default.Table_36c01c2a0d944899bf662092e1cc816d [id=f9add48fee4b4ca89754d81057f832e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.475184 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd5234791524eddb7edd493e39fb993 (DEFAULT_TABLE table=default.Table_20365e1d61b74cb8b39b9a51c2d608c1 [id=2bd0e4026ed5471aa1a457ebea173f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.477734 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 (DEFAULT_TABLE table=default.Table_cba5fd0a90c8433980631ba7e2048d8b [id=3368a0430dc94c4db0ae6260f5a0acf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.478914 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e977f7837e49fb93ef0fd18b619792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.479261 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7945118bb17b431dac8670007d2d0a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.479442 28836 raft_consensus.cc:504] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.480576 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3514b79467dc42bc89fdacf8e39a000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.481514 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.480391 28836 leader_election.cc:288] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.483929 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8b0eed57c4ad1bf6040507f6f8d4c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.484812 28736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:03.432286 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_085095beb9c0496385c83ff148946b68 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 1 got 0
I20240503 14:12:03.486584 28736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:03.487226 28736 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240503 14:12:03.435143 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5264956c580441a6a23c96d39a41ea71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.489636 28836 raft_consensus.cc:2781] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.489733 28217 raft_consensus.cc:2454] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:03.489944 28836 raft_consensus.cc:686] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:03.490234 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 102) took 1049 ms. Trace:
I20240503 14:12:03.490370 28836 consensus_queue.cc:237] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.490419 28653 rpcz_store.cc:276] 0503 14:12:02.413946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.415579 (+ 1633us) service_pool.cc:225] Handling call
0503 14:12:03.028162 (+612583us) ts_tablet_manager.cc:1120] Deleting tablet 5d637087c60144aaaa9b0965110d280e
0503 14:12:03.028174 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.125927 (+ 97753us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.368475 (+242548us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.463725 (+ 95250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":606643}
I20240503 14:12:03.493400 28600 tablet_bootstrap.cc:492] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:03.497136 28492 raft_consensus.cc:2454] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:03.497455 28600 ts_tablet_manager.cc:1367] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.005s
W20240503 14:12:03.498082 28150 leader_election.cc:341] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.498476 28150 leader_election.cc:302] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:03.502439 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 605531ed29214b1ca27011c5f5e96b11 (DEFAULT_TABLE table=default.Table_037096975b83431e99d595ed4e62d706 [id=651246fc28e144af9b8a0bbe9926aa22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.513760 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 651478ef632144079f93f3b50bff4611 (DEFAULT_TABLE table=default.Table_419d8eaa44c34cca86e012471fd9814d [id=ece08717161d44f7987b32a0ccc1cf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.521224 28610 raft_consensus.cc:2781] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.521394 28653 tablet_replica.cc:330] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:03.521482 28610 raft_consensus.cc:482] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.521767 28836 raft_consensus.cc:482] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.522697 28610 raft_consensus.cc:3037] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.523488 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd76060daab84c2f9f6727b8a9bb3a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.528074 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_085095beb9c0496385c83ff148946b68 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 1 got 0
I20240503 14:12:03.528666 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]). Expected version 2 got 0
I20240503 14:12:03.529096 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 2ef6cd66f34c4980bad5e500309b1568 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]). Expected version 1 got 0
I20240503 14:12:03.529662 28027 catalog_manager.cc:5484] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:03.530508 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
I20240503 14:12:03.535537 28653 raft_consensus.cc:2229] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:03.536154 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651478ef632144079f93f3b50bff4611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.538064 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd5234791524eddb7edd493e39fb993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.538452 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605531ed29214b1ca27011c5f5e96b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.536156 28600 raft_consensus.cc:348] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.538955 28600 raft_consensus.cc:374] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.539139 28600 raft_consensus.cc:729] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.539685 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93462d2b1a474bd688d6266cd3b0695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.539543 28600 consensus_queue.cc:260] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.540459 28600 ts_tablet_manager.cc:1398] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.043s user 0.000s sys 0.003s
I20240503 14:12:03.540990 28600 tablet_bootstrap.cc:492] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:03.541095 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265dc5f2630e453ea71abddb1a9753f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.543133 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129717ef137c41c5a08d401c64164187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.543164 28653 raft_consensus.cc:2258] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:03.544754 28653 ts_tablet_manager.cc:1875] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:03.535495 28836 raft_consensus.cc:504] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.552857 28812 raft_consensus.cc:2781] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:12:03.556419 28278 kernel_stack_watchdog.cc:198] Thread 28290 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 209ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f52c1d25980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f52be993d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f52be9b47ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f52be9b03ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f52be9ae84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f52ca1e319b kudu::consensus::RaftConsensus::ElectionCallback() at ??:0
@ 0x7f52ca1eddb4 kudu::consensus::RaftConsensus::StartElection()::$_3::operator()() at ??:0
@ 0x7f52ca1edac2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f52ca151c9e std::function<>::operator()() at ??:0
@ 0x7f52ca14b5b3 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f52ca14c782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f52ca14f12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f52ca14eeee std::_Function_handler<>::_M_invoke() at ??:0
I20240503 14:12:03.563228 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8b0eed57c4ad1bf6040507f6f8d4c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.563408 28715 raft_consensus.cc:504] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.564461 28715 leader_election.cc:288] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:03.565188 28603 ts_tablet_manager.cc:1888] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:03.565382 28603 log.cc:1196] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ec6186c8f2e546f39bf0f632d76166fa
W20240503 14:12:03.565768 28150 leader_election.cc:341] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:03.565912 28603 ts_tablet_manager.cc:1909] T ec6186c8f2e546f39bf0f632d76166fa P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:03.565996 28150 leader_election.cc:302] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:03.568287 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc337647c7cd402e91720478380f4696" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.568709 28812 raft_consensus.cc:482] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.565014 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1de9717d0e44179b875174d5a76e4c3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:03.568934 28427 leader_election.cc:302] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:03.571883 28494 raft_consensus.cc:2454] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:03.575119 28812 raft_consensus.cc:3037] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.577085 28781 raft_consensus.cc:2726] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:03.580293 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 55086.8us
I20240503 14:12:03.580947 28492 raft_consensus.cc:3037] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.586009 28825 raft_consensus.cc:2781] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.588033 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a2bfdcb8e2b241ca9a6922fd8491e27b
I20240503 14:12:03.588886 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc4fe0dac143484396ef862744cc9967" } modify_external_catalogs: true
I20240503 14:12:03.590495 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a8d2600031df4d77b9afe636a9b5ad40
I20240503 14:12:03.592442 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fa5fb9616f9d41748b6c0b3e261e7f24" } new_table_name: "default.Table_61bbab05d19346f4b659073493a0a3c6" modify_external_catalogs: true
W20240503 14:12:03.596191 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f51834695b8b4a729a088da81eee2e47 consensus state has no leader
I20240503 14:12:03.596813 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_21a60d0d25074a54abc69f6309875813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.597541 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21a60d0d25074a54abc69f6309875813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.599313 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_33f62889c4784d6cb42ac76faa20113c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.599999 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33f62889c4784d6cb42ac76faa20113c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.604290 28715 raft_consensus.cc:482] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.605024 28715 raft_consensus.cc:504] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.605293 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8543af24474940c3b1ed368e33cba893" } new_table_name: "default.Table_8c90a0cf9e1f495ea0ea85f1f8726f52" modify_external_catalogs: true
I20240503 14:12:03.607262 28715 leader_election.cc:288] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:03.607946 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7712b0c57d054073ac3f31fa183e72f1" } modify_external_catalogs: true
I20240503 14:12:03.608187 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7712b0c57d054073ac3f31fa183e72f1" } modify_external_catalogs: true
I20240503 14:12:03.613682 28825 raft_consensus.cc:686] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:03.614679 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:03.614872 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 440146us
I20240503 14:12:03.615213 28836 leader_election.cc:288] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.616158 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 605531ed29214b1ca27011c5f5e96b11 (DEFAULT_TABLE table=default.Table_037096975b83431e99d595ed4e62d706 [id=651246fc28e144af9b8a0bbe9926aa22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.616830 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605531ed29214b1ca27011c5f5e96b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.621733 28150 leader_election.cc:302] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:03.624039 28614 tablet_bootstrap.cc:492] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:03.624413 28614 ts_tablet_manager.cc:1367] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.671s user 0.045s sys 0.129s
W20240503 14:12:03.627473 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 48915.7us
I20240503 14:12:03.627722 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 5d637087c60144aaaa9b0965110d280e (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]) successfully deleted
I20240503 14:12:03.628394 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc337647c7cd402e91720478380f4696" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.628847 28354 raft_consensus.cc:2454] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:03.630103 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c800e997e944158f7c839dd0e00d44" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:03.630508 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:03.630872 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 651478ef632144079f93f3b50bff4611 (DEFAULT_TABLE table=default.Table_419d8eaa44c34cca86e012471fd9814d [id=ece08717161d44f7987b32a0ccc1cf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.631372 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651478ef632144079f93f3b50bff4611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.631374 28604 tablet_bootstrap.cc:654] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.632009 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bfdcb8e2b241ca9a6922fd8491e27b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:03.632907 28836 raft_consensus.cc:2781] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.633111 28614 raft_consensus.cc:348] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.633394 28614 raft_consensus.cc:374] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.633514 28614 raft_consensus.cc:729] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.633810 28614 consensus_queue.cc:260] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.637472 28614 ts_tablet_manager.cc:1398] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:03.637926 28614 tablet_bootstrap.cc:492] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:03.640223 28614 tablet_bootstrap.cc:654] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.630872 28825 consensus_queue.cc:237] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.644544 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166fe6366ba496a869af1b106fe9d8e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.652393 28836 raft_consensus.cc:482] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.655148 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 129717ef137c41c5a08d401c64164187 (DEFAULT_TABLE table=default.Table_36c01c2a0d944899bf662092e1cc816d [id=f9add48fee4b4ca89754d81057f832e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.655922 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129717ef137c41c5a08d401c64164187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.652912 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c800e997e944158f7c839dd0e00d44" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:03.657588 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 129717ef137c41c5a08d401c64164187 (DEFAULT_TABLE table=default.Table_36c01c2a0d944899bf662092e1cc816d [id=f9add48fee4b4ca89754d81057f832e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.658107 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129717ef137c41c5a08d401c64164187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.658627 28493 raft_consensus.cc:3037] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.659984 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 93462d2b1a474bd688d6266cd3b0695c (DEFAULT_TABLE table=default.Table_7712b0c57d054073ac3f31fa183e72f1 [id=48c3f3e039044d10a6cfa1b6bcd5ae79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.660785 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93462d2b1a474bd688d6266cd3b0695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.662276 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 81) took 1629 ms. Trace:
I20240503 14:12:03.663661 28492 raft_consensus.cc:2454] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
W20240503 14:12:03.665493 28017 consensus_peers.cc:485] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 -> Peer bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571): Couldn't send request to peer bb6a7aee99214df4b02858ddae025b14. Status: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:03.662572 28603 rpcz_store.cc:276] 0503 14:12:02.032997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.033083 (+ 86us) service_pool.cc:225] Handling call
0503 14:12:02.853944 (+820861us) ts_tablet_manager.cc:1120] Deleting tablet ec6186c8f2e546f39bf0f632d76166fa
0503 14:12:02.853957 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.226776 (+372819us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x6160000d2754. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52ca1dea00 00007f52cb4b8b83 00007f52cdf4c57c 00007f52cdf7e8d4 00007f52cdf5fbc0 00007f52cdf5f94d 0000000000a60bcc 00007f52be9b6b74 00007f52be9ba19b 00007f52be9ba018
0503 14:12:03.369134 (+142358us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.564947 (+195813us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.662210 (+ 97263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":839512832,"tablet-delete.queue_time_us":820513}
I20240503 14:12:03.666877 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 96) took 1280 ms. Trace:
I20240503 14:12:03.667572 28603 rpcz_store.cc:276] 0503 14:12:02.385965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.386049 (+ 84us) service_pool.cc:225] Handling call
0503 14:12:03.666615 (+1280566us) ts_tablet_manager.cc:1120] Deleting tablet b1de9717d0e44179b875174d5a76e4c3
0503 14:12:03.666632 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.666807 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280235}
I20240503 14:12:03.668180 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 97) took 1280 ms. Trace:
I20240503 14:12:03.668855 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 93462d2b1a474bd688d6266cd3b0695c (DEFAULT_TABLE table=default.Table_7712b0c57d054073ac3f31fa183e72f1 [id=48c3f3e039044d10a6cfa1b6bcd5ae79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.669375 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 (DEFAULT_TABLE table=default.Table_cba5fd0a90c8433980631ba7e2048d8b [id=3368a0430dc94c4db0ae6260f5a0acf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.669445 28217 raft_consensus.cc:2454] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:03.669744 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166fe6366ba496a869af1b106fe9d8e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:03.669766 28603 rpcz_store.cc:276] 0503 14:12:02.387280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.387339 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:03.667962 (+1280623us) ts_tablet_manager.cc:1120] Deleting tablet a0e8b0eed57c4ad1bf6040507f6f8d4c
0503 14:12:03.667973 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.668124 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280358}
I20240503 14:12:03.670257 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 100) took 1275 ms. Trace:
I20240503 14:12:03.670501 28603 rpcz_store.cc:276] 0503 14:12:02.394379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.394889 (+ 510us) service_pool.cc:225] Handling call
0503 14:12:03.670075 (+1275186us) ts_tablet_manager.cc:1120] Deleting tablet e85b6db1a21c4bca80c8bdec17e45c59
0503 14:12:03.670085 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.670204 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1273637}
I20240503 14:12:03.671097 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93462d2b1a474bd688d6266cd3b0695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.673540 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_79726aaee899484aaf8147ce414cbd87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.674297 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79726aaee899484aaf8147ce414cbd87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.674378 28337 tablet_service.cc:1467] Processing CreateTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 (DEFAULT_TABLE table=default.Table_cef464f155254da3a6e7a33965f98e2d [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.675184 28603 tablet_replica.cc:330] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:03.675993 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 605531ed29214b1ca27011c5f5e96b11 (DEFAULT_TABLE table=default.Table_037096975b83431e99d595ed4e62d706 [id=651246fc28e144af9b8a0bbe9926aa22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.676402 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 7945118bb17b431dac8670007d2d0a18 (DEFAULT_TABLE table=default.Table_e145076fa64d4661bf830b556d2d2351 [id=892862b894c547e384ebf8b33b9a7ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.676013 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd5234791524eddb7edd493e39fb993 (DEFAULT_TABLE table=default.Table_20365e1d61b74cb8b39b9a51c2d608c1 [id=2bd0e4026ed5471aa1a457ebea173f93]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:03.676949 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b5992e28f1745678618230c183609e4 consensus state has no leader
I20240503 14:12:03.677489 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_642763ad4fae43e2b4f0d0c764202168"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.677685 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265dc5f2630e453ea71abddb1a9753f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:03.678141 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_642763ad4fae43e2b4f0d0c764202168 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.679919 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7945118bb17b431dac8670007d2d0a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.680557 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5ddacd70c694443e8f6e5639e1ea2afc" } new_table_name: "default.Table_6add22bd3ca9452fa98a991f4e9b817f" modify_external_catalogs: true
I20240503 14:12:03.681135 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 63e977f7837e49fb93ef0fd18b619792 (DEFAULT_TABLE table=default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4 [id=dcdb4c8fe4774779b0303db262dbfcc8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:03.682766 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54f08aac9a7147ebaffda4262a57a86a has no consensus state
I20240503 14:12:03.683219 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_17a1019962da48c0992b5452d4a67aae" } modify_external_catalogs: true
I20240503 14:12:03.683480 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_17a1019962da48c0992b5452d4a67aae" } modify_external_catalogs: true
I20240503 14:12:03.683579 28603 raft_consensus.cc:2229] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:03.683874 28333 tablet_service.cc:1467] Processing CreateTablet for tablet dda39f0323ec47799d3d44c96f3ba605 (DEFAULT_TABLE table=default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.684631 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 (DEFAULT_TABLE table=default.Table_068cea3b090241c7980d0edb57c05aec [id=80b934d1638444ebb855ebacf917c23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.685000 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd76060daab84c2f9f6727b8a9bb3a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.686931 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet eb79edba14b646ef9be239b05a291d71 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:03.687414 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7de7832122804eada175f710cc5bab5e
I20240503 14:12:03.687500 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 3514b79467dc42bc89fdacf8e39a000d (DEFAULT_TABLE table=default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.688197 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet bacf2d1091ee458c961c691093ea65a9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:03.688364 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda39f0323ec47799d3d44c96f3ba605. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:03.688802 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 779f373ef12a4da091309fcf247b5693 consensus state has no leader
I20240503 14:12:03.689121 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet b1de9717d0e44179b875174d5a76e4c3 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:03.689795 28603 pending_rounds.cc:62] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:03.694392 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd5234791524eddb7edd493e39fb993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.694581 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 93462d2b1a474bd688d6266cd3b0695c
I20240503 14:12:03.695291 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet a2bfdcb8e2b241ca9a6922fd8491e27b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:03.695868 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 2ef6cd66f34c4980bad5e500309b1568 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:03.695776 28715 consensus_queue.cc:1035] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:12:03.696259 28603 pending_rounds.cc:68] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597660906192896 op_type: NO_OP noop_request { }
I20240503 14:12:03.696442 28603 raft_consensus.cc:2866] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:03.696591 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3514b79467dc42bc89fdacf8e39a000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.696911 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:03.696924 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_106726a25eb74a22951f48bee14b67dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.697662 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_106726a25eb74a22951f48bee14b67dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.698171 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bfdcb8e2b241ca9a6922fd8491e27b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:03.699512 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9d8f6693297844bc907fc355a5561203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.700121 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d8f6693297844bc907fc355a5561203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.705842 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.709250 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605531ed29214b1ca27011c5f5e96b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.710525 28836 raft_consensus.cc:3037] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.708945 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c800e997e944158f7c839dd0e00d44" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:03.712343 28214 raft_consensus.cc:3037] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.712432 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e977f7837e49fb93ef0fd18b619792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.712559 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab05178dc081480ba409aab7f143f849" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:03.713218 28603 raft_consensus.cc:2258] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:03.713575 28354 raft_consensus.cc:3037] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.715950 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2502a13e8cbc4a3abfed225f94c44d2f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:03.716256 28352 raft_consensus.cc:3037] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.716652 28150 leader_election.cc:302] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:03.717096 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 52727.6us
I20240503 14:12:03.717581 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c875ec8c5e49d1bb20a23dd7a21902" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:03.717845 28216 raft_consensus.cc:2454] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:03.721127 28603 ts_tablet_manager.cc:1875] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:03.721999 28781 raft_consensus.cc:2781] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.722659 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23186a7a0dfb432e813c8a05e97e46dd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:03.723610 28357 raft_consensus.cc:2454] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:03.736794 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f3fb3fa343fe469c9f4da86b992bd15a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:03.737639 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3fb3fa343fe469c9f4da86b992bd15a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.739694 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_719b0269379f4b64b84d65d8b9bb3f69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.740845 28839 consensus_queue.cc:1035] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.741536 28839 consensus_queue.cc:1035] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.742488 28600 tablet_bootstrap.cc:654] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:03.744915 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_719b0269379f4b64b84d65d8b9bb3f69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.749097 28290 leader_election.cc:302] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:03.760965 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 651478ef632144079f93f3b50bff4611 (DEFAULT_TABLE table=default.Table_419d8eaa44c34cca86e012471fd9814d [id=ece08717161d44f7987b32a0ccc1cf36]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:03.761811 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 95959.4us
I20240503 14:12:03.778050 28839 raft_consensus.cc:2726] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:03.787906 28290 leader_election.cc:302] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:03.789490 28352 raft_consensus.cc:2454] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:03.789916 28839 raft_consensus.cc:2781] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.793148 28839 raft_consensus.cc:482] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.793352 28839 raft_consensus.cc:3037] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.801483 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 (DEFAULT_TABLE table=default.Table_cba5fd0a90c8433980631ba7e2048d8b [id=3368a0430dc94c4db0ae6260f5a0acf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.656987 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c875ec8c5e49d1bb20a23dd7a21902" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:03.803534 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7832122804eada175f710cc5bab5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:03.804234 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 93462d2b1a474bd688d6266cd3b0695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:03.804302 28290 consensus_peers.cc:485] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d637087c60144aaaa9b0965110d280e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:03.804666 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:03.805644 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b1de9717d0e44179b875174d5a76e4c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1de9717d0e44179b875174d5a76e4c3 already in progress: creating tablet
I20240503 14:12:03.805671 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet ab05178dc081480ba409aab7f143f849 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:03.805996 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e8b0eed57c4ad1bf6040507f6f8d4c already in progress: creating tablet
W20240503 14:12:03.806264 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e85b6db1a21c4bca80c8bdec17e45c59 because the tablet was not found. No further retry: Not found: Tablet not found: e85b6db1a21c4bca80c8bdec17e45c59
I20240503 14:12:03.807493 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651478ef632144079f93f3b50bff4611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.809420 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 (DEFAULT_TABLE table=default.Table_068cea3b090241c7980d0edb57c05aec [id=80b934d1638444ebb855ebacf917c23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.813627 28214 raft_consensus.cc:2454] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:03.815085 28604 tablet_bootstrap.cc:492] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:03.817085 28491 raft_consensus.cc:2454] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:03.816015 28614 tablet_bootstrap.cc:492] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:03.817929 28614 ts_tablet_manager.cc:1367] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.180s user 0.007s sys 0.015s
I20240503 14:12:03.818545 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd5234791524eddb7edd493e39fb993 (DEFAULT_TABLE table=default.Table_20365e1d61b74cb8b39b9a51c2d608c1 [id=2bd0e4026ed5471aa1a457ebea173f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.819468 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 7945118bb17b431dac8670007d2d0a18 (DEFAULT_TABLE table=default.Table_e145076fa64d4661bf830b556d2d2351 [id=892862b894c547e384ebf8b33b9a7ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.822217 28459 tablet_service.cc:1467] Processing CreateTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 (DEFAULT_TABLE table=default.Table_cef464f155254da3a6e7a33965f98e2d [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.824754 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265dc5f2630e453ea71abddb1a9753f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.825301 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 63e977f7837e49fb93ef0fd18b619792 (DEFAULT_TABLE table=default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4 [id=dcdb4c8fe4774779b0303db262dbfcc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.827247 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 3514b79467dc42bc89fdacf8e39a000d (DEFAULT_TABLE table=default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.826025 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 93462d2b1a474bd688d6266cd3b0695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:03.827778 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7832122804eada175f710cc5bab5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:03.827687 28614 raft_consensus.cc:348] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.828294 28614 raft_consensus.cc:374] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.828444 28614 raft_consensus.cc:729] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.828501 28781 raft_consensus.cc:686] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:03.829025 28604 ts_tablet_manager.cc:1367] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.465s user 0.021s sys 0.031s
I20240503 14:12:03.830091 28614 consensus_queue.cc:260] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.831879 28614 ts_tablet_manager.cc:1398] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240503 14:12:03.840481 28754 raft_consensus.cc:482] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.840801 28869 raft_consensus.cc:482] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.841104 28869 raft_consensus.cc:504] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:03.832432 28781 consensus_queue.cc:237] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.842361 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:03.843797 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4b5992e28f1745678618230c183609e4 consensus state has no leader
I20240503 14:12:03.845752 28869 leader_election.cc:288] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:03.848723 28614 tablet_bootstrap.cc:492] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:03.843204 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502a13e8cbc4a3abfed225f94c44d2f with delete_type TABLET_DATA_DELETED (replaced by tablet da19787034c24ebe929601d49cb21b3b) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.855082 28754 raft_consensus.cc:504] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:03.855662 28610 raft_consensus.cc:504] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.856056 28754 leader_election.cc:288] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.856393 28610 leader_election.cc:288] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:03.855281 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 56268.7us
I20240503 14:12:03.860975 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cba5fd0a90c8433980631ba7e2048d8b" } modify_external_catalogs: true
I20240503 14:12:03.861279 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cba5fd0a90c8433980631ba7e2048d8b" } modify_external_catalogs: true
I20240503 14:12:03.861913 28354 raft_consensus.cc:2454] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:03.864557 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c6f41bb9c5d40b0913264864ecc3528 has no consensus state
I20240503 14:12:03.865231 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5a5d61e70c514fbfac570c0d594e7010" } new_table_name: "default.Table_466e18e3e6f346e798971d5b48a74866" modify_external_catalogs: true
I20240503 14:12:03.866986 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45eb624434a404d85dcd233721ebf68" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:03.867169 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b166fe6366ba496a869af1b106fe9d8e
I20240503 14:12:04.270258 28927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.254:36571 every 8 connection(s)
I20240503 14:12:03.875326 28873 raft_consensus.cc:482] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.270328 28736 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.254:36571
I20240503 14:12:04.321638 28873 raft_consensus.cc:504] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.876883 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:04.322613 28873 leader_election.cc:288] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:03.873052 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8c90a0cf9e1f495ea0ea85f1f8726f52" } modify_external_catalogs: true
I20240503 14:12:03.878291 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2ef6cd66f34c4980bad5e500309b1568
I20240503 14:12:03.894209 28456 tablet_service.cc:1467] Processing CreateTablet for tablet dda39f0323ec47799d3d44c96f3ba605 (DEFAULT_TABLE table=default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.322993 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8c90a0cf9e1f495ea0ea85f1f8726f52" } modify_external_catalogs: true
I20240503 14:12:04.323462 28033 catalog_manager.cc:6554] ReplaceTablet: tablet 2ef6cd66f34c4980bad5e500309b1568 deleted and replaced by tablet a7295c2d3bd34e20a69986aadc9c7017
I20240503 14:12:04.323998 28736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
I20240503 14:12:03.899513 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet b166fe6366ba496a869af1b106fe9d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:03.893951 28614 tablet_bootstrap.cc:654] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.905650 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f3fb3fa343fe469c9f4da86b992bd15a" } new_table_name: "default.Table_40f2cb0b89114bc291d2474f9babca16" modify_external_catalogs: true
I20240503 14:12:03.902967 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef6cd66f34c4980bad5e500309b1568 with delete_type TABLET_DATA_DELETED (replaced by tablet a7295c2d3bd34e20a69986aadc9c7017) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:03.962474 28603 ts_tablet_manager.cc:1888] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:04.325161 28603 log.cc:1196] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6106d52df3c54c2bb4abadd8605cd912
I20240503 14:12:04.325538 28192 tablet_service.cc:1467] Processing CreateTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 (DEFAULT_TABLE table=default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.325670 28603 ts_tablet_manager.cc:1909] T 6106d52df3c54c2bb4abadd8605cd912 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:03.897193 28604 raft_consensus.cc:348] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.325904 28604 raft_consensus.cc:374] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.325990 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7295c2d3bd34e20a69986aadc9c7017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.326035 28604 raft_consensus.cc:729] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.326050 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 101) took 1931 ms. Trace:
I20240503 14:12:04.326154 28603 rpcz_store.cc:276] 0503 14:12:02.394505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.394559 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:03.670902 (+1276343us) ts_tablet_manager.cc:1120] Deleting tablet 6106d52df3c54c2bb4abadd8605cd912
0503 14:12:03.670913 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.856425 (+185512us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.962248 (+105823us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.326004 (+363756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274213}
I20240503 14:12:04.326305 28604 consensus_queue.cc:260] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.326517 28603 tablet_replica.cc:330] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:03.901666 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7945118bb17b431dac8670007d2d0a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.326748 28603 raft_consensus.cc:2229] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:04.327108 28603 raft_consensus.cc:2258] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:04.327752 28603 ts_tablet_manager.cc:1875] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:03.963387 28839 raft_consensus.cc:504] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.328708 28839 leader_election.cc:288] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:03.880944 28290 leader_election.cc:302] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
W20240503 14:12:04.329049 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 448205us
W20240503 14:12:04.194700 28278 kernel_stack_watchdog.cc:198] Thread 28290 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 113ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f52c1d25980 (unknown) at ??:0
@ 0x7f52c1d1f9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f52bbca5904 google::LogMessage::Flush() at ??:0
@ 0x7f52bbca5a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f52ca14b3f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f52ca14c782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f52ca14f12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f52ca14eeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f52c55e58cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f52c55e54ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f52c55d14e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f52c55d1a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f52c556e51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f52c556cf45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240503 14:12:04.329838 28842 raft_consensus.cc:2781] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.907532 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_00e404138f154498b730db1de48ded05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:03.904933 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c800e997e944158f7c839dd0e00d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:04.330772 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00e404138f154498b730db1de48ded05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.917863 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 559342d357d64f9eb9e3ab42ccb75942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.918145 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:04.331548 28494 raft_consensus.cc:2454] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:03.918608 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet ab05178dc081480ba409aab7f143f849 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.920214 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bfdcb8e2b241ca9a6922fd8491e27b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.918751 28812 raft_consensus.cc:504] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.908895 28600 tablet_bootstrap.cc:492] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:03.927440 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:04.332228 28600 ts_tablet_manager.cc:1367] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.791s user 0.012s sys 0.005s
I20240503 14:12:03.935123 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 93462d2b1a474bd688d6266cd3b0695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.945756 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7832122804eada175f710cc5bab5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.954269 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet b166fe6366ba496a869af1b106fe9d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:03.949812 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45eb624434a404d85dcd233721ebf68" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:03.960845 28493 raft_consensus.cc:2454] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:04.333130 28600 raft_consensus.cc:348] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.333393 28600 raft_consensus.cc:374] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.333482 28600 raft_consensus.cc:729] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.333760 28600 consensus_queue.cc:260] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.334412 28600 ts_tablet_manager.cc:1398] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:04.334775 28600 tablet_bootstrap.cc:492] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:03.908150 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5e7ed1c5e8934a81bfeba19fe7452080"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:04.335449 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e7ed1c5e8934a81bfeba19fe7452080 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:03.962863 28836 raft_consensus.cc:504] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.337185 28836 leader_election.cc:288] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:04.013813 28825 consensus_queue.cc:1035] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.016153 28754 consensus_queue.cc:1035] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.908812 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_719b0269379f4b64b84d65d8b9bb3f69" } new_table_name: "default.Table_151ead7d1ad1474b8facf3f42c1cdac1" modify_external_catalogs: true
I20240503 14:12:04.019376 28427 leader_election.cc:302] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:04.337951 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda39f0323ec47799d3d44c96f3ba605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.964016 28653 ts_tablet_manager.cc:1888] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:04.338898 28653 log.cc:1196] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/559342d357d64f9eb9e3ab42ccb75942
I20240503 14:12:04.339309 28653 ts_tablet_manager.cc:1909] T 559342d357d64f9eb9e3ab42ccb75942 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:04.268491 28416 kernel_stack_watchdog.cc:198] Thread 28427 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 192ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f66fb884980 (unknown) at ??:0
@ 0x7f66fb87e9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f66f5804904 google::LogMessage::Flush() at ??:0
@ 0x7f66f5804a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f6703caa3f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f6703cab782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f6703cae12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f6703cadeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f66ff1448cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f66ff1444ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f66ff1304e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f66ff130a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f66ff0cd51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f66ff0cbf45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240503 14:12:04.340219 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 100) took 1939 ms. Trace:
I20240503 14:12:04.340335 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd76060daab84c2f9f6727b8a9bb3a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.340338 28653 rpcz_store.cc:276] 0503 14:12:02.400868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.400925 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:03.496035 (+1095110us) ts_tablet_manager.cc:1120] Deleting tablet 559342d357d64f9eb9e3ab42ccb75942
0503 14:12:03.496047 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.877283 (+381236us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:03.963813 (+ 86530us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.340174 (+376361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059601}
I20240503 14:12:04.340724 28653 tablet_replica.cc:330] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:04.340934 28653 raft_consensus.cc:2229] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:04.341092 28653 raft_consensus.cc:2258] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:04.341727 28653 ts_tablet_manager.cc:1875] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:04.342546 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3514b79467dc42bc89fdacf8e39a000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.956892 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
I20240503 14:12:03.975019 28781 raft_consensus.cc:482] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.343858 28781 raft_consensus.cc:504] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:04.344672 28781 leader_election.cc:288] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:03.961637 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_33f62889c4784d6cb42ac76faa20113c" } modify_external_catalogs: true
I20240503 14:12:04.345070 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_33f62889c4784d6cb42ac76faa20113c" } modify_external_catalogs: true
I20240503 14:12:03.989161 28868 raft_consensus.cc:482] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.345716 28868 raft_consensus.cc:504] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.346544 28868 leader_election.cc:288] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:04.010963 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 265dc5f2630e453ea71abddb1a9753f3
I20240503 14:12:04.347506 28842 raft_consensus.cc:686] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:04.347617 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7b0e6de295714677a8c47970ae8d75e8" } new_table_name: "default.Table_ac2a630ab8d046f88a48f486ed63dc80" modify_external_catalogs: true
I20240503 14:12:03.990538 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 962df44408484e3f8d7b7138f27c1a66 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.349329 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962df44408484e3f8d7b7138f27c1a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.347898 28842 consensus_queue.cc:237] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:03.991106 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd (DEFAULT_TABLE table=default.Table_652b2cc6b6024e738ecdd3cb52306637 [id=5c194c0fc1c94da0a40f0f3f0d7484b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.352535 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b139ab0b5864024ac48215d46b0a7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.991479 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc (DEFAULT_TABLE table=default.Table_9a954e776a184d819708a62a99ed94fc [id=713e339595f142ab8a82bb724d23a487]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.991858 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 (DEFAULT_TABLE table=default.Table_c9442a77363b44c584467e25ecc9878f [id=19a9b36e090c46a483c355f0f5cd6c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.992393 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 (DEFAULT_TABLE table=default.Table_6add22bd3ca9452fa98a991f4e9b817f [id=806d4f0bc4524d9481487418ff797a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.993029 28186 tablet_service.cc:1467] Processing CreateTablet for tablet ed9262a7e828489ab51a745068c737df (DEFAULT_TABLE table=default.Table_dbb71b03f52c48d29206e945b956f702 [id=fd3e3a26ac344865b4c622ebe130ebff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.993410 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 1d77014c66624f958b38f7cc83644891 (DEFAULT_TABLE table=default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.010838 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:04.356640 28217 raft_consensus.cc:2454] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:04.011128 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 54f08aac9a7147ebaffda4262a57a86a (DEFAULT_TABLE table=default.Table_fc0d4a5f37ae46d9807079682b94ba1d [id=f406220394264c8f824c0e1fce8479e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.066642 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba (DEFAULT_TABLE table=default.Table_f3fb3fa343fe469c9f4da86b992bd15a [id=caca4dbaa701453794ac56475f46704e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.074864 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 1044f34588d045ce898435b61a90503f (DEFAULT_TABLE table=default.Table_9d8f6693297844bc907fc355a5561203 [id=3c43a0fcf51143179a1ed0e1b8e333c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.085523 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (DEFAULT_TABLE table=default.Table_719b0269379f4b64b84d65d8b9bb3f69 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.097013 28184 tablet_service.cc:1467] Processing CreateTablet for tablet f2b861dccd964092ae0f06668c9b58fe (DEFAULT_TABLE table=default.Table_106726a25eb74a22951f48bee14b67dd [id=aaeb9d44d9f04208a57d49b81068662d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.109705 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 (DEFAULT_TABLE table=default.Table_79726aaee899484aaf8147ce414cbd87 [id=8cce76d1e8a642a4b051b4c4a854be78]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:03.888140 28150 leader_election.cc:341] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:04.358063 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 469979us
I20240503 14:12:04.359230 28938 sys_catalog.cc:263] Verifying existing consensus state
I20240503 14:12:04.359778 28150 leader_election.cc:302] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:04.360147 28150 leader_election.cc:341] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:04.138218 28182 tablet_service.cc:1467] Processing CreateTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 (DEFAULT_TABLE table=default.Table_33f62889c4784d6cb42ac76faa20113c [id=c62152292eb14f4aaae391155d12da6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.160743 28181 tablet_service.cc:1467] Processing CreateTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 (DEFAULT_TABLE table=default.Table_e3e1acbcbb7c4aeab3db92cc253b441d [id=9ac543611bee46c08fb673c287aa6af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.174425 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 37dc7d96357743089372baf71654b1e3 (DEFAULT_TABLE table=default.Table_21a60d0d25074a54abc69f6309875813 [id=11af424ea24549bc83f6203448ae619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.175791 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b (DEFAULT_TABLE table=default.Table_5264956c580441a6a23c96d39a41ea71 [id=59887894cf0843df9cd01850035c1e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.176721 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a (DEFAULT_TABLE table=default.Table_642763ad4fae43e2b4f0d0c764202168 [id=d7eac407621f4b03b21439fc2b0f7dc3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:04.243499 28139 kernel_stack_watchdog.cc:198] Thread 28150 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 229ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f2f26cf5980 (unknown) at ??:0
@ 0x7f2f26cef9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f2f20c75904 google::LogMessage::Flush() at ??:0
@ 0x7f2f20c75a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f2f2f11c5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f2f2f11f12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f2f2f11eeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f2f2a5b58cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f2f2a5b54ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f2f2a5a14e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f2f2a5a1a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f2f2a53e51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f2f2a53cf45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f2f2a54f2c5 ev::base<>::method_thunk<>() at ??:0
I20240503 14:12:04.364267 28938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Bootstrap starting.
I20240503 14:12:04.364430 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 28736
I20240503 14:12:04.364377 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1751f775fe6943d9b5fcc3b113393a73" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:04.364976 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc337647c7cd402e91720478380f4696" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:04.365233 28357 raft_consensus.cc:3037] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.366016 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 559342d357d64f9eb9e3ab42ccb75942 (table default.Table_7d8e4d1ecd794871b9d73289f8f43c9a [id=c6805f8e2643425dbed90e5ebe7d3a4a]) successfully deleted
I20240503 14:12:04.367309 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef6cd66f34c4980bad5e500309b1568 with delete_type TABLET_DATA_DELETED (replaced by tablet a7295c2d3bd34e20a69986aadc9c7017) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:04.367595 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:04.368376 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19787034c24ebe929601d49cb21b3b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:04.368875 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4775ff1029e6449c83219030e3809984" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:04.369220 28868 raft_consensus.cc:2781] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:04.369357 28868 raft_consensus.cc:482] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:04.369446 28868 raft_consensus.cc:3037] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:04.373374 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a32b6995818041778617a0d6bf936c8c consensus state has no leader
I20240503 14:12:04.373881 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dbb71b03f52c48d29206e945b956f702" } modify_external_catalogs: true
I20240503 14:12:04.374125 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dbb71b03f52c48d29206e945b956f702" } modify_external_catalogs: true
I20240503 14:12:04.376168 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 779f373ef12a4da091309fcf247b5693
I20240503 14:12:04.376801 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_bbc0f1621f4842c2a8b5913acd913703"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:04.376955 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_852b77a0ef2e4ee6bda7be721348aff8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:04.377451 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc0f1621f4842c2a8b5913acd913703 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:04.379807 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 779f373ef12a4da091309fcf247b5693 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:04.381464 28150 leader_election.cc:341] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:04.381765 28150 leader_election.cc:341] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:04.381984 28150 leader_election.cc:341] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:04.387276 28812 leader_election.cc:288] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:04.387754 28490 raft_consensus.cc:3037] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.390767 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc337647c7cd402e91720478380f4696" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:04.391041 28493 raft_consensus.cc:3037] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.392761 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19787034c24ebe929601d49cb21b3b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:04.393446 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1751f775fe6943d9b5fcc3b113393a73" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:04.394161 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50edc38cdbe74397aa4f4928d56d9fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.394848 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9262a7e828489ab51a745068c737df. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:04.377451 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_852b77a0ef2e4ee6bda7be721348aff8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:04.395391 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6f41bb9c5d40b0913264864ecc3528. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.395845 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d77014c66624f958b38f7cc83644891. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.396832 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4775ff1029e6449c83219030e3809984" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:04.399673 28825 raft_consensus.cc:2781] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:04.399909 28825 raft_consensus.cc:686] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:04.400274 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:04.400393 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 379006us
I20240503 14:12:04.402441 28825 consensus_queue.cc:237] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.405297 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.406248 28427 leader_election.cc:302] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:04.407189 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd (DEFAULT_TABLE table=default.Table_652b2cc6b6024e738ecdd3cb52306637 [id=5c194c0fc1c94da0a40f0f3f0d7484b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.407748 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b139ab0b5864024ac48215d46b0a7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.407858 28754 raft_consensus.cc:482] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a64bc15d65d84ad5a005960a9a84df9c)
I20240503 14:12:04.408077 28754 raft_consensus.cc:504] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:04.408274 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1044f34588d045ce898435b61a90503f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.408952 28754 leader_election.cc:288] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:04.409198 28754 raft_consensus.cc:2781] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:04.409297 28754 raft_consensus.cc:482] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:04.409432 28754 raft_consensus.cc:3037] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.412425 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3703ab7db61b4c9ead3058bfdc123189. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:04.413132 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 83719.8us
W20240503 14:12:04.413587 28290 consensus_peers.cc:485] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:04.416069 28839 raft_consensus.cc:482] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.416328 28839 raft_consensus.cc:504] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.416597 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f08aac9a7147ebaffda4262a57a86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.417102 28839 leader_election.cc:288] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:04.419642 28812 consensus_queue.cc:1035] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240503 14:12:04.420555 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004d556f3ed44a7db1a52d205ecfd90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.421443 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 1044f34588d045ce898435b61a90503f (DEFAULT_TABLE table=default.Table_9d8f6693297844bc907fc355a5561203 [id=3c43a0fcf51143179a1ed0e1b8e333c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.421936 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1044f34588d045ce898435b61a90503f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.424366 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dc7d96357743089372baf71654b1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.424419 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 54f08aac9a7147ebaffda4262a57a86a (DEFAULT_TABLE table=default.Table_fc0d4a5f37ae46d9807079682b94ba1d [id=f406220394264c8f824c0e1fce8479e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.426379 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc (DEFAULT_TABLE table=default.Table_9a954e776a184d819708a62a99ed94fc [id=713e339595f142ab8a82bb724d23a487]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.428279 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6371e68ee85a47cdb0e8e21f7124d94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.412994 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 1d77014c66624f958b38f7cc83644891 (DEFAULT_TABLE table=default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.428177 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 (DEFAULT_TABLE table=default.Table_6add22bd3ca9452fa98a991f4e9b817f [id=806d4f0bc4524d9481487418ff797a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.432358 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0982b99a02ee48f2b6379e7be56c4c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.435521 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:04.431696 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c875ec8c5e49d1bb20a23dd7a21902" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:04.435695 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba (DEFAULT_TABLE table=default.Table_f3fb3fa343fe469c9f4da86b992bd15a [id=caca4dbaa701453794ac56475f46704e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.435940 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b861dccd964092ae0f06668c9b58fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.425187 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 962df44408484e3f8d7b7138f27c1a66 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.437072 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dbb59d98834ab6bbac4abfa92681e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.440348 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6627f0d9a544bb8176bea6c1af5ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.442404 28472 tablet_service.cc:1467] Processing CreateTablet for tablet ed9262a7e828489ab51a745068c737df (DEFAULT_TABLE table=default.Table_dbb71b03f52c48d29206e945b956f702 [id=fd3e3a26ac344865b4c622ebe130ebff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.445351 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c875ec8c5e49d1bb20a23dd7a21902" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:04.445694 28217 raft_consensus.cc:3037] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.448196 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f08aac9a7147ebaffda4262a57a86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.449465 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 (DEFAULT_TABLE table=default.Table_c9442a77363b44c584467e25ecc9878f [id=19a9b36e090c46a483c355f0f5cd6c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.450800 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (DEFAULT_TABLE table=default.Table_719b0269379f4b64b84d65d8b9bb3f69 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.451304 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.452558 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23186a7a0dfb432e813c8a05e97e46dd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:04.453225 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23186a7a0dfb432e813c8a05e97e46dd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:04.453516 28353 raft_consensus.cc:3037] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.454218 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9262a7e828489ab51a745068c737df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.454689 28491 raft_consensus.cc:3037] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.455143 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6f41bb9c5d40b0913264864ecc3528. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.455883 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet b166fe6366ba496a869af1b106fe9d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:04.456523 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd (DEFAULT_TABLE table=default.Table_652b2cc6b6024e738ecdd3cb52306637 [id=5c194c0fc1c94da0a40f0f3f0d7484b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.456683 28469 tablet_service.cc:1467] Processing CreateTablet for tablet f2b861dccd964092ae0f06668c9b58fe (DEFAULT_TABLE table=default.Table_106726a25eb74a22951f48bee14b67dd [id=aaeb9d44d9f04208a57d49b81068662d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.457432 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d77014c66624f958b38f7cc83644891. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.457538 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc (DEFAULT_TABLE table=default.Table_9a954e776a184d819708a62a99ed94fc [id=713e339595f142ab8a82bb724d23a487]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.458149 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962df44408484e3f8d7b7138f27c1a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.458149 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 54f08aac9a7147ebaffda4262a57a86a (DEFAULT_TABLE table=default.Table_fc0d4a5f37ae46d9807079682b94ba1d [id=f406220394264c8f824c0e1fce8479e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.458655 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 1d77014c66624f958b38f7cc83644891 (DEFAULT_TABLE table=default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.458825 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50edc38cdbe74397aa4f4928d56d9fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.459143 28337 tablet_service.cc:1467] Processing CreateTablet for tablet ed9262a7e828489ab51a745068c737df (DEFAULT_TABLE table=default.Table_dbb71b03f52c48d29206e945b956f702 [id=fd3e3a26ac344865b4c622ebe130ebff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.459645 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 (DEFAULT_TABLE table=default.Table_c9442a77363b44c584467e25ecc9878f [id=19a9b36e090c46a483c355f0f5cd6c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.460220 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 (DEFAULT_TABLE table=default.Table_6add22bd3ca9452fa98a991f4e9b817f [id=806d4f0bc4524d9481487418ff797a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.461505 28329 tablet_service.cc:1467] Processing CreateTablet for tablet f2b861dccd964092ae0f06668c9b58fe (DEFAULT_TABLE table=default.Table_106726a25eb74a22951f48bee14b67dd [id=aaeb9d44d9f04208a57d49b81068662d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.464291 28661 ts_tablet_manager.cc:1888] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:04.464610 28661 log.cc:1196] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/567f15be6b9e4baaba9587c3634f72ad
I20240503 14:12:04.465023 28661 ts_tablet_manager.cc:1909] T 567f15be6b9e4baaba9587c3634f72ad P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:04.465307 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b861dccd964092ae0f06668c9b58fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.465353 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (DEFAULT_TABLE table=default.Table_719b0269379f4b64b84d65d8b9bb3f69 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.465757 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 (DEFAULT_TABLE table=default.Table_79726aaee899484aaf8147ce414cbd87 [id=8cce76d1e8a642a4b051b4c4a854be78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.465930 28357 raft_consensus.cc:2454] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:04.465970 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.466522 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3703ab7db61b4c9ead3058bfdc123189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.466545 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 962df44408484e3f8d7b7138f27c1a66 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.467176 28324 tablet_service.cc:1467] Processing CreateTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 (DEFAULT_TABLE table=default.Table_33f62889c4784d6cb42ac76faa20113c [id=c62152292eb14f4aaae391155d12da6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.456549 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba (DEFAULT_TABLE table=default.Table_f3fb3fa343fe469c9f4da86b992bd15a [id=caca4dbaa701453794ac56475f46704e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.468935 28214 raft_consensus.cc:3037] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.469853 28323 tablet_service.cc:1467] Processing CreateTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 (DEFAULT_TABLE table=default.Table_e3e1acbcbb7c4aeab3db92cc253b441d [id=9ac543611bee46c08fb673c287aa6af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.470964 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 1044f34588d045ce898435b61a90503f (DEFAULT_TABLE table=default.Table_9d8f6693297844bc907fc355a5561203 [id=3c43a0fcf51143179a1ed0e1b8e333c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.471804 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b (DEFAULT_TABLE table=default.Table_5264956c580441a6a23c96d39a41ea71 [id=59887894cf0843df9cd01850035c1e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.474421 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e977f7837e49fb93ef0fd18b619792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.476795 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a (DEFAULT_TABLE table=default.Table_642763ad4fae43e2b4f0d0c764202168 [id=d7eac407621f4b03b21439fc2b0f7dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.482518 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef6cd66f34c4980bad5e500309b1568 with delete_type TABLET_DATA_DELETED (replaced by tablet a7295c2d3bd34e20a69986aadc9c7017) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:04.483059 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:04.476928 28319 tablet_service.cc:1467] Processing CreateTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 (DEFAULT_TABLE table=default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.486380 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 37dc7d96357743089372baf71654b1e3 (DEFAULT_TABLE table=default.Table_21a60d0d25074a54abc69f6309875813 [id=11af424ea24549bc83f6203448ae619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.488243 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 779f373ef12a4da091309fcf247b5693 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:04.491813 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet b1de9717d0e44179b875174d5a76e4c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a32b6995818041778617a0d6bf936c8c) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:04.497493 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd5234791524eddb7edd493e39fb993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.498798 28456 tablet_service.cc:1467] Processing CreateTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 (DEFAULT_TABLE table=default.Table_e3e1acbcbb7c4aeab3db92cc253b441d [id=9ac543611bee46c08fb673c287aa6af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.502372 28461 tablet_service.cc:1467] Processing CreateTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 (DEFAULT_TABLE table=default.Table_33f62889c4784d6cb42ac76faa20113c [id=c62152292eb14f4aaae391155d12da6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.503764 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 (DEFAULT_TABLE table=default.Table_79726aaee899484aaf8147ce414cbd87 [id=8cce76d1e8a642a4b051b4c4a854be78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.503697 28956 consensus_queue.cc:1035] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:04.504276 28150 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 82340.6us
I20240503 14:12:04.504508 28956 consensus_queue.cc:1035] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.505535 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0982b99a02ee48f2b6379e7be56c4c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.505780 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962df44408484e3f8d7b7138f27c1a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.506310 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9262a7e828489ab51a745068c737df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.506619 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b861dccd964092ae0f06668c9b58fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.506937 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.507012 28836 raft_consensus.cc:482] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.507558 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d77014c66624f958b38f7cc83644891. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:04.507838 28150 consensus_peers.cc:485] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:04.508165 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3703ab7db61b4c9ead3058bfdc123189. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:04.508404 28150 leader_election.cc:341] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:04.508706 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f08aac9a7147ebaffda4262a57a86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.508883 28150 leader_election.cc:302] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:04.509222 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6f41bb9c5d40b0913264864ecc3528. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.509346 28873 raft_consensus.cc:2726] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:04.509724 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50edc38cdbe74397aa4f4928d56d9fbc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:04.509825 28150 leader_election.cc:341] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:04.510294 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dbb59d98834ab6bbac4abfa92681e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.511405 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.511950 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b139ab0b5864024ac48215d46b0a7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.512530 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1044f34588d045ce898435b61a90503f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.513199 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6627f0d9a544bb8176bea6c1af5ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.513727 28493 raft_consensus.cc:2454] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:04.513820 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004d556f3ed44a7db1a52d205ecfd90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.514375 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7295c2d3bd34e20a69986aadc9c7017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.514992 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6371e68ee85a47cdb0e8e21f7124d94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.515673 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dc7d96357743089372baf71654b1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.524374 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dbb59d98834ab6bbac4abfa92681e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.530019 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 37dc7d96357743089372baf71654b1e3 (DEFAULT_TABLE table=default.Table_21a60d0d25074a54abc69f6309875813 [id=11af424ea24549bc83f6203448ae619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.530468 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0982b99a02ee48f2b6379e7be56c4c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.531929 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dc7d96357743089372baf71654b1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.535620 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b (DEFAULT_TABLE table=default.Table_5264956c580441a6a23c96d39a41ea71 [id=59887894cf0843df9cd01850035c1e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.537434 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6627f0d9a544bb8176bea6c1af5ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.537990 28603 ts_tablet_manager.cc:1888] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:12:04.672808 28938 log.cc:824] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:04.582130 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6106d52df3c54c2bb4abadd8605cd912 (table default.Table_6270d949a6654e54b3cc252a54943112 [id=ada8758d8af14ac7bc31969934efb7d1]) successfully deleted
W20240503 14:12:04.764536 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 182406us
W20240503 14:12:04.768709 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba consensus state has no leader
W20240503 14:12:04.802629 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 consensus state has no leader
W20240503 14:12:04.804222 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f03bf91cf5414208b6767482223bc240 consensus state has no leader
I20240503 14:12:04.805744 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d3937b734026471d8da300845379e2dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:04.806442 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3937b734026471d8da300845379e2dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:04.810068 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_976081c8371b4dc68cb226625ef1b6c6" } new_table_name: "default.Table_afc737ad630d4de6b239f807771fa6a1" modify_external_catalogs: true
I20240503 14:12:04.813553 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2e0a149bc29c4f2fac86f41074f45afa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:04.813840 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6fb64d577a8944049a29de7ed12a6255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:04.814263 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e0a149bc29c4f2fac86f41074f45afa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:04.569936 28150 leader_election.cc:302] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:04.928372 28781 raft_consensus.cc:2726] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:04.577160 28868 raft_consensus.cc:504] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:04.929623 28868 leader_election.cc:288] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:04.586499 28600 tablet_bootstrap.cc:654] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.541612 28603 log.cc:1196] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/5d637087c60144aaaa9b0965110d280e
I20240503 14:12:04.570165 28614 tablet_bootstrap.cc:492] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:04.930109 28614 ts_tablet_manager.cc:1367] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 1.081s user 0.006s sys 0.021s
I20240503 14:12:04.930327 28603 ts_tablet_manager.cc:1909] T 5d637087c60144aaaa9b0965110d280e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:04.930778 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 99) took 2536 ms. Trace:
I20240503 14:12:04.930891 28603 rpcz_store.cc:276] 0503 14:12:02.394256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.394929 (+ 673us) service_pool.cc:225] Handling call
0503 14:12:04.326450 (+1931521us) ts_tablet_manager.cc:1120] Deleting tablet 5d637087c60144aaaa9b0965110d280e
0503 14:12:04.326460 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:04.386612 (+ 60152us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.521024 (+134412us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.930703 (+409679us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929503}
I20240503 14:12:04.931226 28614 raft_consensus.cc:348] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:04.618023 28353 raft_consensus.cc:2454] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:04.931458 28603 tablet_replica.cc:330] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:04.679205 28951 tablet.cc:1722] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:04.931699 28603 raft_consensus.cc:2229] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:04.931902 28603 pending_rounds.cc:62] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:04.730311 28943 tablet.cc:1722] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:04.931962 28603 pending_rounds.cc:68] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597660783169536 op_type: NO_OP noop_request { }
I20240503 14:12:04.543411 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004d556f3ed44a7db1a52d205ecfd90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.830098 28980 tablet.cc:1722] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:04.546818 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 72) took 2636 ms. Trace:
I20240503 14:12:04.684463 28214 raft_consensus.cc:2454] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:04.694825 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1de9717d0e44179b875174d5a76e4c3" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:04.932261 28661 rpcz_store.cc:276] 0503 14:12:01.908406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:01.909672 (+ 1266us) service_pool.cc:225] Handling call
0503 14:12:03.230589 (+1320917us) ts_tablet_manager.cc:1120] Deleting tablet 567f15be6b9e4baaba9587c3634f72ad
0503 14:12:03.230604 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:03.993829 (+763225us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.464135 (+470306us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.544827 (+ 80692us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320695}
I20240503 14:12:04.762836 28653 ts_tablet_manager.cc:1888] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:04.804911 28217 raft_consensus.cc:2454] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:04.932708 28653 log.cc:1196] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a0e8b0eed57c4ad1bf6040507f6f8d4c
I20240503 14:12:04.932727 28661 tablet_replica.cc:330] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:04.832772 28403 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:04.575018 28491 raft_consensus.cc:2454] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:04.933135 28653 ts_tablet_manager.cc:1909] T a0e8b0eed57c4ad1bf6040507f6f8d4c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:04.576380 28455 tablet_service.cc:1467] Processing CreateTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 (DEFAULT_TABLE table=default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.933565 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7295c2d3bd34e20a69986aadc9c7017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.934047 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 101) took 2533 ms. Trace:
I20240503 14:12:04.934162 28653 rpcz_store.cc:276] 0503 14:12:02.400975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.401078 (+ 103us) service_pool.cc:225] Handling call
0503 14:12:04.340654 (+1939576us) ts_tablet_manager.cc:1120] Deleting tablet a0e8b0eed57c4ad1bf6040507f6f8d4c
0503 14:12:04.340666 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:04.664717 (+324051us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.762656 (+ 97939us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.933980 (+171324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1907858}
I20240503 14:12:04.934587 28653 tablet_replica.cc:330] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:04.934787 28653 raft_consensus.cc:2229] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:04.934974 28653 raft_consensus.cc:2258] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:04.935622 28653 ts_tablet_manager.cc:1875] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:04.579442 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a (DEFAULT_TABLE table=default.Table_642763ad4fae43e2b4f0d0c764202168 [id=d7eac407621f4b03b21439fc2b0f7dc3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:04.928474 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 418744us
I20240503 14:12:04.608310 28939 consensus_queue.cc:1035] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
W20240503 14:12:04.936831 28150 leader_election.cc:341] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:04.936954 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6371e68ee85a47cdb0e8e21f7124d94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.608734 28604 ts_tablet_manager.cc:1398] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.751s user 0.008s sys 0.004s
I20240503 14:12:04.937002 28150 leader_election.cc:302] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:04.937166 28604 ts_tablet_manager.cc:1430] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Tablet startup took 1107ms
I20240503 14:12:04.937505 28150 leader_election.cc:302] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:04.816129 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7ad492405f92420ba1dd8dcf793d915c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:04.931483 28614 raft_consensus.cc:374] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.941123 28603 raft_consensus.cc:2866] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
W20240503 14:12:04.814263 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fb64d577a8944049a29de7ed12a6255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.055927 28614 raft_consensus.cc:729] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.056034 28603 raft_consensus.cc:2258] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:05.056416 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ad492405f92420ba1dd8dcf793d915c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.056376 28614 consensus_queue.cc:260] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:05.057088 28614 ts_tablet_manager.cc:1398] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.127s user 0.003s sys 0.000s
I20240503 14:12:04.836544 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a6dbb59d98834ab6bbac4abfa92681e3
I20240503 14:12:04.942116 28355 raft_consensus.cc:3037] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.057511 28614 tablet_bootstrap.cc:492] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:04.943213 28290 leader_election.cc:302] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:05.057890 28030 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 220) took 1151 ms. Trace:
W20240503 14:12:05.057932 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 121369us
I20240503 14:12:05.058943 28956 raft_consensus.cc:2781] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:04.962643 28938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Bootstrap replayed 1/1 log segments. Stats: ops{read=41 overwritten=0 applied=41 ignored=0} inserts{seen=43 ignored=0} mutations{seen=27 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:04.938022 28868 raft_consensus.cc:2726] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:04.938230 28781 raft_consensus.cc:2781] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.060492 28781 raft_consensus.cc:686] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:05.060948 28938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Bootstrap complete.
I20240503 14:12:05.060834 28781 consensus_queue.cc:237] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.573578 28836 raft_consensus.cc:504] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.062819 28836 leader_election.cc:288] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.059139 28956 raft_consensus.cc:686] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:04.978900 28943 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58028 (request call id 146) took 1295 ms. Trace:
I20240503 14:12:05.059863 28603 ts_tablet_manager.cc:1875] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:04.940061 28263 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:05.065018 28956 consensus_queue.cc:237] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:04.943640 28873 consensus_queue.cc:1035] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.058019 28030 rpcz_store.cc:276] 0503 14:12:03.906781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.961451 (+ 54670us) service_pool.cc:225] Handling call
0503 14:12:04.345301 (+383850us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:04.345470 (+ 169us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:04.367738 (+ 22268us) catalog_manager.cc:2758] Locking tablets
0503 14:12:04.367965 (+ 227us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:04.836196 (+468231us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:04.836306 (+ 110us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:04.836352 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:05.057827 (+221475us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:04.369103 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:04.369228 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:04.369234 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:04.369236 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:04.369369 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:04.369383 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:04.369389 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:04.369534 (+ 145us) tablet.cc:699] Row locks acquired
0503 14:12:04.369537 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:04.369585 (+ 48us) write_op.cc:270] Start()
0503 14:12:04.369627 (+ 42us) write_op.cc:276] Timestamp: P: 1714745524369533 usec, L: 0
0503 14:12:04.369632 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:04.369845 (+ 213us) log.cc:842] Serialized 731 byte log entry
0503 14:12:04.805308 (+435463us) op_driver.cc:444] REPLICATION: finished
0503 14:12:04.825810 (+ 20502us) write_op.cc:301] APPLY: starting
0503 14:12:04.825836 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:04.825898 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:04.825976 (+ 78us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:04.826087 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:04.826136 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:04.826145 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:04.826261 (+ 116us) log.cc:842] Serialized 29 byte log entry
0503 14:12:04.827166 (+ 905us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:04.827196 (+ 30us) write_op.cc:454] Released schema lock
0503 14:12:04.827207 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:04.836765 (+ 9558us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:04.836775 (+ 10us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":98,"child_traces":[["op",{"apply.queue_time_us":20391,"apply.run_cpu_time_us":838,"apply.run_wall_time_us":11073,"num_ops":2,"prepare.queue_time_us":256,"prepare.run_cpu_time_us":857,"prepare.run_wall_time_us":857,"raft.queue_time_us":540,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":435652}]]}
I20240503 14:12:04.964118 29004 consensus_queue.cc:1035] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:12:04.980065 28653 ts_tablet_manager.cc:1888] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:04.937335 28604 ts_tablet_manager.cc:1432] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Trace:
0503 14:12:01.763337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:01.763397 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:01.763859 (+ 462us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:01.763914 (+ 55us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:01.772883 (+ 8969us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:01.844916 (+ 72033us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:01.845127 (+ 211us) inbound_call.cc:173] Queueing success response
0503 14:12:03.350451 (+1505324us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 14:12:03.829843 (+479392us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 14:12:03.834687 (+ 4844us) tablet_replica.cc:259] Starting instrumentation
0503 14:12:03.835029 (+ 342us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":11095,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":44723,"tablet-open.queue_time_us":1505381,"thread_start_us":216455,"threads_started":1}
I20240503 14:12:05.066601 28653 log.cc:1196] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/92c800e997e944158f7c839dd0e00d44
I20240503 14:12:04.688460 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1de9717d0e44179b875174d5a76e4c3" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:05.068074 28604 tablet_bootstrap.cc:492] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:05.068339 28653 ts_tablet_manager.cc:1909] T 92c800e997e944158f7c839dd0e00d44 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:04.768474 28754 raft_consensus.cc:504] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:04.982784 28600 tablet_bootstrap.cc:492] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:05.068691 28600 ts_tablet_manager.cc:1367] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.734s user 0.007s sys 0.007s
I20240503 14:12:05.069227 28754 leader_election.cc:288] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.069581 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e355aafb2f5b4337bedaac6fe5539dc2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:05.069927 28604 tablet_bootstrap.cc:654] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.069756 28600 raft_consensus.cc:348] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.070041 28600 raft_consensus.cc:374] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.070192 28600 raft_consensus.cc:729] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.070725 28600 consensus_queue.cc:260] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.066702 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4" } modify_external_catalogs: true
I20240503 14:12:05.075548 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4" } modify_external_catalogs: true
I20240503 14:12:05.075963 28600 ts_tablet_manager.cc:1398] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:05.076380 28600 tablet_bootstrap.cc:492] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:04.943812 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:05.076902 28354 raft_consensus.cc:3037] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.871912 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]). Expected version 2 got 0
I20240503 14:12:05.077486 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet f45eb624434a404d85dcd233721ebf68 (table default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842]). Expected version 1 got 0
I20240503 14:12:05.064795 28943 rpcz_store.cc:276] 0503 14:12:03.683440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.683491 (+ 51us) service_pool.cc:225] Handling call
0503 14:12:04.978787 (+1295296us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:03.685245 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:03.688509 (+ 3264us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:03.688519 (+ 10us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:03.688527 (+ 8us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:03.688580 (+ 53us) alter_schema_op.cc:119] START. Timestamp: 1714745523688526
0503 14:12:03.688584 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:03.691269 (+ 2685us) log.cc:842] Serialized 173 byte log entry
0503 14:12:04.347718 (+656449us) op_driver.cc:444] REPLICATION: finished
0503 14:12:04.727721 (+380003us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:04.943659 (+215938us) log.cc:842] Serialized 5 byte log entry
0503 14:12:04.974691 (+ 31032us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.978682 (+ 3991us) log.cc:842] Serialized 14 byte log entry
0503 14:12:04.978744 (+ 62us) alter_schema_op.cc:65] Released schema lock
0503 14:12:04.978748 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"spinlock_wait_cycles":1243008,"child_traces":[["op",{"apply.queue_time_us":353107,"prepare.queue_time_us":75,"prepare.run_cpu_time_us":1283,"prepare.run_wall_time_us":6138,"raft.queue_time_us":488,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":2861,"replication_time_us":659100,"thread_start_us":334812,"threads_started":2,"wal-append.queue_time_us":10939}]]}
I20240503 14:12:04.883708 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
I20240503 14:12:05.078616 28600 tablet_bootstrap.cc:654] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.072458 28980 tablet.cc:1717] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:05.079258 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 93462d2b1a474bd688d6266cd3b0695c from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:05.079872 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 1693dd36a3d84d88a49690c537590083 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:05.071002 28150 leader_election.cc:341] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:05.070186 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e355aafb2f5b4337bedaac6fe5539dc2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:04.771800 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 779f373ef12a4da091309fcf247b5693 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:04.771117 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef6cd66f34c4980bad5e500309b1568 with delete_type TABLET_DATA_DELETED (replaced by tablet a7295c2d3bd34e20a69986aadc9c7017) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:04.774885 28460 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51314 (request call id 131) took 1135 ms. Trace:
I20240503 14:12:05.082609 28460 rpcz_store.cc:276] 0503 14:12:03.639789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.639857 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:03.818940 (+179083us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:03.819011 (+ 71us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:04.605443 (+786432us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.774694 (+169251us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:04.774812 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 14:12:04.851532 28984 tablet.cc:1722] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240503 14:12:05.080283 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23186a7a0dfb432e813c8a05e97e46dd consensus state has no leader
I20240503 14:12:05.083089 28028 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 228) took 1003 ms. Trace:
W20240503 14:12:05.083127 28150 leader_election.cc:341] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:04.583959 28427 leader_election.cc:341] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 3826594d300342c7a59e0eee989a873f
I20240503 14:12:04.666479 28490 raft_consensus.cc:2454] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
W20240503 14:12:05.083444 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 500341us
I20240503 14:12:05.083539 28490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.27.50.193:45137 (request call id 48) took 1156 ms. Trace:
I20240503 14:12:05.075184 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 99) took 2674 ms. Trace:
I20240503 14:12:05.083719 28490 rpcz_store.cc:276] 0503 14:12:03.934223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.936188 (+ 1965us) service_pool.cc:225] Handling call
0503 14:12:05.083475 (+1147287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 14:12:04.940380 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:05.084164 28489 raft_consensus.cc:2379] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:05.084213 28951 tablet.cc:1717] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 3, tablet already version 3
W20240503 14:12:05.085346 28427 leader_election.cc:341] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:05.085888 28939 raft_consensus.cc:482] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.944867 28661 raft_consensus.cc:2229] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.086088 28939 raft_consensus.cc:504] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.083199 28028 rpcz_store.cc:276] 0503 14:12:04.079330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.809818 (+730488us) service_pool.cc:225] Handling call
0503 14:12:04.810324 (+ 506us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:04.810520 (+ 196us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:04.810736 (+ 216us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:04.810824 (+ 88us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:04.901630 (+ 90806us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:05.083026 (+181396us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:04.816644 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:04.816764 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:04.816770 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:04.816773 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:04.816886 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:04.816899 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:04.816904 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:04.816986 (+ 82us) tablet.cc:699] Row locks acquired
0503 14:12:04.816989 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:04.817030 (+ 41us) write_op.cc:270] Start()
0503 14:12:04.817070 (+ 40us) write_op.cc:276] Timestamp: P: 1714745524816966 usec, L: 0
0503 14:12:04.817073 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:04.817288 (+ 215us) log.cc:842] Serialized 405 byte log entry
0503 14:12:04.900053 (+ 82765us) op_driver.cc:444] REPLICATION: finished
0503 14:12:04.900238 (+ 185us) write_op.cc:301] APPLY: starting
0503 14:12:04.900254 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:04.900290 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:04.900325 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:04.900374 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:04.900411 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:04.900418 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:04.900512 (+ 94us) log.cc:842] Serialized 23 byte log entry
0503 14:12:04.900579 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:04.900598 (+ 19us) write_op.cc:454] Released schema lock
0503 14:12:04.900605 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:04.900666 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:04.900669 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":484,"mutex_wait_us":76,"num_ops":1,"prepare.queue_time_us":5256,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":731,"replication_time_us":82963}]]}
I20240503 14:12:05.086341 28661 raft_consensus.cc:2258] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:04.932952 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 5d637087c60144aaaa9b0965110d280e (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]) successfully deleted
W20240503 14:12:05.086594 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 153601us
I20240503 14:12:05.086863 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet f45eb624434a404d85dcd233721ebf68 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:05.086935 28939 leader_election.cc:288] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.087108 28661 ts_tablet_manager.cc:1875] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.087517 28939 consensus_queue.cc:1035] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:04.856149 28019 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 54f08aac9a7147ebaffda4262a57a86a (table default.Table_fc0d4a5f37ae46d9807079682b94ba1d [id=f406220394264c8f824c0e1fce8479e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54f08aac9a7147ebaffda4262a57a86a consensus state has no leader
I20240503 14:12:05.088446 28150 leader_election.cc:302] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:05.089237 28150 leader_election.cc:302] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:04.870303 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ed9262a7e828489ab51a745068c737df
I20240503 14:12:05.090255 28025 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 222) took 1183 ms. Trace:
I20240503 14:12:05.090370 28025 rpcz_store.cc:276] 0503 14:12:03.907048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.373710 (+466662us) service_pool.cc:225] Handling call
0503 14:12:04.374230 (+ 520us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:04.374365 (+ 135us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:04.374436 (+ 71us) catalog_manager.cc:2758] Locking tablets
0503 14:12:04.374591 (+ 155us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:04.870047 (+495456us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:04.870181 (+ 134us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:04.870234 (+ 53us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:05.090187 (+219953us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:04.378590 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:04.378701 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:04.378708 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:04.378710 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:04.378838 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:04.378851 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:04.378857 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:04.378984 (+ 127us) tablet.cc:699] Row locks acquired
0503 14:12:04.378986 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:04.379040 (+ 54us) write_op.cc:270] Start()
0503 14:12:04.379079 (+ 39us) write_op.cc:276] Timestamp: P: 1714745524378980 usec, L: 0
0503 14:12:04.379082 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:04.379293 (+ 211us) log.cc:842] Serialized 731 byte log entry
0503 14:12:04.868123 (+488830us) op_driver.cc:444] REPLICATION: finished
0503 14:12:04.868682 (+ 559us) write_op.cc:301] APPLY: starting
0503 14:12:04.868702 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:04.868754 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:04.868784 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:04.868856 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:04.868892 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:04.868900 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:04.868993 (+ 93us) log.cc:842] Serialized 29 byte log entry
0503 14:12:04.869062 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:04.869083 (+ 21us) write_op.cc:454] Released schema lock
0503 14:12:04.869091 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:04.869146 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:04.869150 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":491,"apply.run_cpu_time_us":532,"apply.run_wall_time_us":532,"num_ops":2,"prepare.queue_time_us":3279,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":811,"raft.queue_time_us":807,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":488977}]]}
I20240503 14:12:05.091460 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_36c01c2a0d944899bf662092e1cc816d" } modify_external_catalogs: true
I20240503 14:12:05.091723 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_36c01c2a0d944899bf662092e1cc816d" } modify_external_catalogs: true
I20240503 14:12:04.901499 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d7bd421e71724c8a89b7cd478f447a3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.092595 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7bd421e71724c8a89b7cd478f447a3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.080933 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
I20240503 14:12:05.081691 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet bd76060daab84c2f9f6727b8a9bb3a49 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:04.902202 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b76d17fed23441d3acf329eaa3d64853"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.091089 28938 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
W20240503 14:12:05.096005 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b76d17fed23441d3acf329eaa3d64853 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.096302 28938 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6a7aee99214df4b02858ddae025b14, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.097501 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.097585 28781 consensus_queue.cc:1035] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20240503 14:12:05.099963 28427 leader_election.cc:341] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 3826594d300342c7a59e0eee989a873f
I20240503 14:12:05.100217 28427 leader_election.cc:302] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:05.099952 28938 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
W20240503 14:12:05.100565 28427 raft_consensus.cc:2670] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:05.101411 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_085095beb9c0496385c83ff148946b68" } new_table_name: "default.Table_ddcf99aec7384d15b0149124ff7a6646" modify_external_catalogs: true
I20240503 14:12:05.102124 28427 leader_election.cc:302] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:05.104457 28825 raft_consensus.cc:2781] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.104686 28825 raft_consensus.cc:686] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:05.105089 28825 consensus_queue.cc:237] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.105198 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 567f15be6b9e4baaba9587c3634f72ad (table default.Table_666d072a7d2a4affa01a3cf64abf777c [id=c094b16f94d44132bea89dc256965971]) successfully deleted
I20240503 14:12:05.106369 28614 tablet_bootstrap.cc:654] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.108433 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:05.108887 28214 raft_consensus.cc:2379] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:05.097385 28653 rpcz_store.cc:276] 0503 14:12:02.400652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.401050 (+ 398us) service_pool.cc:225] Handling call
0503 14:12:04.934495 (+2533445us) ts_tablet_manager.cc:1120] Deleting tablet 92c800e997e944158f7c839dd0e00d44
0503 14:12:04.934506 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:04.970436 (+ 35930us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:04.975950 (+ 5514us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.075142 (+ 99192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2501113}
I20240503 14:12:05.112525 28653 tablet_replica.cc:330] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:05.113389 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.113574 28540 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
W20240503 14:12:05.014457 28416 kernel_stack_watchdog.cc:198] Thread 28427 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 340ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f66fb884980 (unknown) at ??:0
@ 0x7f66fb87e9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f66f5804904 google::LogMessage::Flush() at ??:0
@ 0x7f66f5804a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f6703cab5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f6703cae12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f6703cadeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f66ff1448cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f66ff1444ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f66ff1304e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f66ff130a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f66ff0cd51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f66ff0cbf45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f66ff0de2c5 ev::base<>::method_thunk<>() at ??:0
I20240503 14:12:04.771117 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.068028 28493 raft_consensus.cc:2379] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:05.107739 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a0e8b0eed57c4ad1bf6040507f6f8d4c (table default.Table_4033322e04a3488cbc8be0fe6a61bec1 [id=e3f646d9aa8946529e897c83c0b10f44]) successfully deleted
I20240503 14:12:05.120018 28604 tablet_bootstrap.cc:492] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:05.120321 28604 ts_tablet_manager.cc:1367] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.002s
I20240503 14:12:05.121585 28604 raft_consensus.cc:348] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.121927 28604 raft_consensus.cc:374] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.122043 28604 raft_consensus.cc:729] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.122366 28604 consensus_queue.cc:260] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.122997 28604 ts_tablet_manager.cc:1398] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:05.123440 28604 tablet_bootstrap.cc:492] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:05.126996 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 92c800e997e944158f7c839dd0e00d44 (table default.Table_5f10f98658754315a6cc9b1d7ffc19cb [id=d9b865dc744148df9fc1ab6e79c5043d]) successfully deleted
I20240503 14:12:05.128979 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9262a7e828489ab51a745068c737df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.129832 28031 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 229) took 1050 ms. Trace:
I20240503 14:12:05.130025 28031 rpcz_store.cc:276] 0503 14:12:04.079736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.813561 (+733825us) service_pool.cc:225] Handling call
0503 14:12:05.056062 (+242501us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:05.056681 (+ 619us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:05.129443 (+ 72762us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:05.129597 (+ 154us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:05.129715 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:05.059656 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:05.059786 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:05.059793 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:05.059796 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:05.059905 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:05.059920 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:05.059925 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:05.060013 (+ 88us) tablet.cc:699] Row locks acquired
0503 14:12:05.060015 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:05.060064 (+ 49us) write_op.cc:270] Start()
0503 14:12:05.060134 (+ 70us) write_op.cc:276] Timestamp: P: 1714745525060039 usec, L: 0
0503 14:12:05.060138 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:05.060379 (+ 241us) log.cc:842] Serialized 433 byte log entry
0503 14:12:05.127642 (+ 67263us) op_driver.cc:444] REPLICATION: finished
0503 14:12:05.127848 (+ 206us) write_op.cc:301] APPLY: starting
0503 14:12:05.127872 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:05.127929 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:05.127965 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:05.128246 (+ 281us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:05.128292 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:05.128301 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:05.128434 (+ 133us) log.cc:842] Serialized 29 byte log entry
0503 14:12:05.128541 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.128568 (+ 27us) write_op.cc:454] Released schema lock
0503 14:12:05.128577 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:05.128650 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.128653 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":4736,"child_traces":[["op",{"apply.queue_time_us":147,"apply.run_cpu_time_us":870,"apply.run_wall_time_us":869,"num_ops":2,"prepare.queue_time_us":2341,"prepare.run_cpu_time_us":858,"prepare.run_wall_time_us":859,"raft.queue_time_us":1815,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":310,"replication_time_us":67468}]]}
I20240503 14:12:05.131142 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6add22bd3ca9452fa98a991f4e9b817f" } modify_external_catalogs: true
I20240503 14:12:05.131384 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6add22bd3ca9452fa98a991f4e9b817f" } modify_external_catalogs: true
I20240503 14:12:05.131647 28938 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:05.129832 28027 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 226) took 1050 ms. Trace:
I20240503 14:12:05.132853 28027 rpcz_store.cc:276] 0503 14:12:04.079075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.802903 (+723828us) service_pool.cc:225] Handling call
0503 14:12:04.816012 (+ 13109us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:05.056563 (+240551us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:05.056975 (+ 412us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:05.129630 (+ 72655us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:05.129708 (+ 78us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:05.129761 (+ 53us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:05.060549 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:05.060682 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:05.060689 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:05.060691 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:05.060781 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:05.060793 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:05.060797 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:05.060876 (+ 79us) tablet.cc:699] Row locks acquired
0503 14:12:05.060879 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:05.060918 (+ 39us) write_op.cc:270] Start()
0503 14:12:05.060951 (+ 33us) write_op.cc:276] Timestamp: P: 1714745525060895 usec, L: 0
0503 14:12:05.060955 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:05.061153 (+ 198us) log.cc:842] Serialized 433 byte log entry
0503 14:12:05.127736 (+ 66583us) op_driver.cc:444] REPLICATION: finished
0503 14:12:05.128763 (+ 1027us) write_op.cc:301] APPLY: starting
0503 14:12:05.128779 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:05.128835 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:05.128895 (+ 60us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:05.129004 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:05.129036 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:05.129043 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:05.129119 (+ 76us) log.cc:842] Serialized 29 byte log entry
0503 14:12:05.129187 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.129208 (+ 21us) write_op.cc:454] Released schema lock
0503 14:12:05.129215 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:05.129274 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.129278 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":29451136,"child_traces":[["op",{"apply.queue_time_us":992,"apply.run_cpu_time_us":569,"apply.run_wall_time_us":569,"num_ops":2,"prepare.queue_time_us":3215,"prepare.run_cpu_time_us":699,"prepare.run_wall_time_us":698,"raft.queue_time_us":1425,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":66765}]]}
I20240503 14:12:05.133950 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_20365e1d61b74cb8b39b9a51c2d608c1" } new_table_name: "default.Table_40cc648a463747e68942c60a9d687deb" modify_external_catalogs: true
I20240503 14:12:05.138528 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7832122804eada175f710cc5bab5e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:05.141191 28873 raft_consensus.cc:2781] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.141330 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef6cd66f34c4980bad5e500309b1568" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:05.141530 29004 raft_consensus.cc:2726] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:05.142241 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.144372 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f45eb624434a404d85dcd233721ebf68
I20240503 14:12:05.144556 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef6cd66f34c4980bad5e500309b1568" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:05.145030 28028 catalog_manager.cc:6554] ReplaceTablet: tablet f45eb624434a404d85dcd233721ebf68 deleted and replaced by tablet 5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:05.145215 28873 raft_consensus.cc:686] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:05.145594 28873 consensus_queue.cc:237] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.145879 28024 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 230) took 1059 ms. Trace:
I20240503 14:12:05.146004 28024 rpcz_store.cc:276] 0503 14:12:04.086864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.901168 (+814304us) service_pool.cc:225] Handling call
0503 14:12:05.092710 (+191542us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:05.093234 (+ 524us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:05.145552 (+ 52318us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:05.145675 (+ 123us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:05.145771 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:05.096711 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:05.096829 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:05.096836 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:05.096839 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:05.096932 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:05.096947 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:05.096952 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:05.097147 (+ 195us) tablet.cc:699] Row locks acquired
0503 14:12:05.097149 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:05.097194 (+ 45us) write_op.cc:270] Start()
0503 14:12:05.097238 (+ 44us) write_op.cc:276] Timestamp: P: 1714745525097169 usec, L: 0
0503 14:12:05.097243 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:05.097454 (+ 211us) log.cc:842] Serialized 433 byte log entry
0503 14:12:05.141021 (+ 43567us) op_driver.cc:444] REPLICATION: finished
0503 14:12:05.142470 (+ 1449us) write_op.cc:301] APPLY: starting
0503 14:12:05.142481 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:05.142516 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:05.142550 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:05.142658 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:05.142688 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:05.142695 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:05.142749 (+ 54us) log.cc:842] Serialized 29 byte log entry
0503 14:12:05.142788 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.142802 (+ 14us) write_op.cc:454] Released schema lock
0503 14:12:05.142808 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:05.142857 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.142861 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1418,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":434,"num_ops":2,"prepare.queue_time_us":2842,"prepare.run_cpu_time_us":837,"prepare.run_wall_time_us":838,"replication_time_us":43765}]]}
I20240503 14:12:05.147079 28653 raft_consensus.cc:2229] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.147223 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet bd76060daab84c2f9f6727b8a9bb3a49
I20240503 14:12:05.147744 28033 catalog_manager.cc:6554] ReplaceTablet: tablet bd76060daab84c2f9f6727b8a9bb3a49 deleted and replaced by tablet 06fb79fcf30d485798c64c815e372dc1
I20240503 14:12:05.148022 28290 leader_election.cc:302] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:05.148633 28653 pending_rounds.cc:62] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:05.148651 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 63e977f7837e49fb93ef0fd18b619792
I20240503 14:12:05.149183 28030 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 232) took 1060 ms. Trace:
I20240503 14:12:05.149426 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9262a7e828489ab51a745068c737df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.149761 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d637087c60144aaaa9b0965110d280e with delete_type TABLET_DATA_DELETED (replaced by tablet e9609112657f427cbeeb3ef8f28cddf0) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.149339 28030 rpcz_store.cc:276] 0503 14:12:04.088352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.066485 (+978133us) service_pool.cc:225] Handling call
0503 14:12:05.075713 (+ 9228us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:05.075832 (+ 119us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:05.075912 (+ 80us) catalog_manager.cc:2758] Locking tablets
0503 14:12:05.076043 (+ 131us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:05.143850 (+ 67807us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:05.143983 (+ 133us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:05.144041 (+ 58us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:05.149111 (+ 5070us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:05.076817 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:05.076941 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:05.076950 (+ 9us) write_op.cc:435] Acquired schema lock
0503 14:12:05.076952 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:05.077100 (+ 148us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:05.077114 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:05.077119 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:05.077324 (+ 205us) tablet.cc:699] Row locks acquired
0503 14:12:05.077327 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:05.077382 (+ 55us) write_op.cc:270] Start()
0503 14:12:05.077422 (+ 40us) write_op.cc:276] Timestamp: P: 1714745525077355 usec, L: 0
0503 14:12:05.077426 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:05.077666 (+ 240us) log.cc:842] Serialized 731 byte log entry
0503 14:12:05.140907 (+ 63241us) op_driver.cc:444] REPLICATION: finished
0503 14:12:05.141256 (+ 349us) write_op.cc:301] APPLY: starting
0503 14:12:05.141279 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:05.141335 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:05.141368 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:05.141450 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:05.141491 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:05.141500 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:05.141603 (+ 103us) log.cc:842] Serialized 29 byte log entry
0503 14:12:05.141681 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.141708 (+ 27us) write_op.cc:454] Released schema lock
0503 14:12:05.141717 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:05.141781 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.141785 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4320,"child_traces":[["op",{"apply.queue_time_us":284,"apply.run_cpu_time_us":593,"apply.run_wall_time_us":592,"num_ops":2,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":941,"prepare.run_wall_time_us":942,"replication_time_us":63444}]]}
I20240503 14:12:05.149873 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9262a7e828489ab51a745068c737df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:05.150454 28290 consensus_peers.cc:485] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92c800e997e944158f7c839dd0e00d44. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:05.150398 28653 pending_rounds.cc:68] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597660783169536 op_type: NO_OP noop_request { }
I20240503 14:12:05.150594 28653 raft_consensus.cc:2866] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:05.150794 28653 raft_consensus.cc:2258] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.152523 28653 ts_tablet_manager.cc:1875] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.154801 28026 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 231) took 1067 ms. Trace:
I20240503 14:12:05.154929 28026 rpcz_store.cc:276] 0503 14:12:04.087376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.901948 (+814572us) service_pool.cc:225] Handling call
0503 14:12:05.096171 (+194223us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:05.096705 (+ 534us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:05.154415 (+ 57710us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:05.154568 (+ 153us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:05.154702 (+ 134us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:05.098348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:05.098439 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:05.098445 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:05.098448 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:05.098525 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:05.098534 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:05.098538 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:05.098606 (+ 68us) tablet.cc:699] Row locks acquired
0503 14:12:05.098609 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:05.098642 (+ 33us) write_op.cc:270] Start()
0503 14:12:05.098670 (+ 28us) write_op.cc:276] Timestamp: P: 1714745525098619 usec, L: 0
0503 14:12:05.098673 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:05.098826 (+ 153us) log.cc:842] Serialized 433 byte log entry
0503 14:12:05.152633 (+ 53807us) op_driver.cc:444] REPLICATION: finished
0503 14:12:05.152810 (+ 177us) write_op.cc:301] APPLY: starting
0503 14:12:05.152827 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:05.152877 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:05.152930 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:05.153036 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:05.153094 (+ 58us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:05.153102 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:05.153212 (+ 110us) log.cc:842] Serialized 29 byte log entry
0503 14:12:05.153331 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.153351 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:05.153359 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:05.153420 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.153424 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":668,"apply.run_wall_time_us":667,"num_ops":2,"prepare.queue_time_us":1169,"prepare.run_cpu_time_us":532,"prepare.run_wall_time_us":532,"replication_time_us":53941}]]}
W20240503 14:12:05.155817 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13c875ec8c5e49d1bb20a23dd7a21902 consensus state has no leader
I20240503 14:12:05.158025 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0c010da29c3645e6a22acf46e3a01775"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.158769 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c010da29c3645e6a22acf46e3a01775 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.158809 28956 raft_consensus.cc:2726] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:05.160070 28427 consensus_peers.cc:485] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:05.160523 28939 consensus_queue.cc:1035] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.160516 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_627b41e76476426da74d6722301bc59f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.161190 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_627b41e76476426da74d6722301bc59f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.162322 28825 consensus_queue.cc:1035] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.162770 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_652b2cc6b6024e738ecdd3cb52306637" } new_table_name: "default.Table_52932a5c26744566bfc744e2ed1d8aee" modify_external_catalogs: true
I20240503 14:12:05.160528 28427 leader_election.cc:302] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:05.163846 28427 leader_election.cc:302] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:05.164366 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2fbf7b7a2fb34274a2a21b61197964d7" } new_table_name: "default.Table_6d9dd77419724e2a9b5b1efeea5eb053" modify_external_catalogs: true
I20240503 14:12:05.165491 28939 raft_consensus.cc:2726] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:05.165738 28939 raft_consensus.cc:2726] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:05.123754 29027 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:05.166788 29027 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:05.171768 28956 consensus_queue.cc:1035] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.176777 28603 ts_tablet_manager.cc:1888] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.177032 28603 log.cc:1196] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/eb79edba14b646ef9be239b05a291d71
I20240503 14:12:05.177574 28603 ts_tablet_manager.cc:1909] T eb79edba14b646ef9be239b05a291d71 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:05.177943 28354 raft_consensus.cc:2454] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
W20240503 14:12:05.179162 28427 consensus_peers.cc:485] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:05.179575 28604 tablet_bootstrap.cc:654] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.180652 28150 leader_election.cc:302] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:05.181289 28836 raft_consensus.cc:2781] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.181459 28836 raft_consensus.cc:686] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:05.181800 28836 consensus_queue.cc:237] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:05.205730 28604 tablet_bootstrap.cc:492] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:05.209020 28604 ts_tablet_manager.cc:1367] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.002s
I20240503 14:12:05.217948 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 8bfc56c4591c437da5b9686b726686b2 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:05.224979 28604 raft_consensus.cc:348] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.228354 28604 raft_consensus.cc:374] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.229925 28604 raft_consensus.cc:729] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.231844 28661 ts_tablet_manager.cc:1888] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:05.231868 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 102) took 2746 ms. Trace:
I20240503 14:12:05.232008 28661 log.cc:1196] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/3826594d300342c7a59e0eee989a873f
I20240503 14:12:05.232125 28604 consensus_queue.cc:260] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.232123 28603 rpcz_store.cc:276] 0503 14:12:02.485489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.486054 (+ 565us) service_pool.cc:225] Handling call
0503 14:12:04.931343 (+2445289us) ts_tablet_manager.cc:1120] Deleting tablet eb79edba14b646ef9be239b05a291d71
0503 14:12:04.931356 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.101410 (+170054us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.176578 (+ 75168us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.231753 (+ 55175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2444988}
I20240503 14:12:05.232558 28661 ts_tablet_manager.cc:1909] T 3826594d300342c7a59e0eee989a873f P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.232815 28604 ts_tablet_manager.cc:1398] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20240503 14:12:05.232946 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 76) took 3275 ms. Trace:
I20240503 14:12:05.232990 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 104) took 2605 ms. Trace:
I20240503 14:12:05.233096 28603 rpcz_store.cc:276] 0503 14:12:02.627695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.627756 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:05.232678 (+2604922us) ts_tablet_manager.cc:1120] Deleting tablet eee989ff083041dc8dcd2cd1b220780d
0503 14:12:05.232690 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.232938 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2604575}
I20240503 14:12:05.233068 28661 rpcz_store.cc:276] 0503 14:12:01.957202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:01.957274 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:04.932612 (+2975338us) ts_tablet_manager.cc:1120] Deleting tablet 3826594d300342c7a59e0eee989a873f
0503 14:12:04.932623 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.174305 (+241682us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.231664 (+ 57359us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.232903 (+ 1239us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11820,"tablet-delete.queue_time_us":2974958}
I20240503 14:12:05.233577 28603 tablet_replica.cc:330] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:05.233800 28603 raft_consensus.cc:2229] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:05.234263 28603 raft_consensus.cc:2258] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.235059 28603 ts_tablet_manager.cc:1875] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.235296 28661 tablet_replica.cc:330] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.235548 28661 raft_consensus.cc:2229] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.235731 28661 raft_consensus.cc:2258] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.236505 28661 ts_tablet_manager.cc:1875] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.244647 28604 tablet_bootstrap.cc:492] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:05.246929 28604 tablet_bootstrap.cc:654] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.256455 28661 ts_tablet_manager.cc:1888] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.256713 28661 log.cc:1196] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ec6186c8f2e546f39bf0f632d76166fa
I20240503 14:12:05.257215 28661 ts_tablet_manager.cc:1909] T ec6186c8f2e546f39bf0f632d76166fa P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.257628 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 77) took 3228 ms. Trace:
I20240503 14:12:05.257742 28661 rpcz_store.cc:276] 0503 14:12:02.029151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.029209 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:05.233376 (+3204167us) ts_tablet_manager.cc:1120] Deleting tablet ec6186c8f2e546f39bf0f632d76166fa
0503 14:12:05.235196 (+ 1820us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.240328 (+ 5132us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.256259 (+ 15931us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.257562 (+ 1303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3203725}
I20240503 14:12:05.258261 28661 tablet_replica.cc:330] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.258504 28661 raft_consensus.cc:2229] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.258776 28661 raft_consensus.cc:2258] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.259208 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet f45eb624434a404d85dcd233721ebf68 with delete_type TABLET_DATA_DELETED (replaced by tablet 5775006e9f8f4612aa70dfb316de9ea1) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.259764 28661 ts_tablet_manager.cc:1875] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.259855 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet f45eb624434a404d85dcd233721ebf68 with delete_type TABLET_DATA_DELETED (replaced by tablet 5775006e9f8f4612aa70dfb316de9ea1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.260100 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet f45eb624434a404d85dcd233721ebf68 with delete_type TABLET_DATA_DELETED (replaced by tablet 5775006e9f8f4612aa70dfb316de9ea1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.260491 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fb79fcf30d485798c64c815e372dc1) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.261503 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fb79fcf30d485798c64c815e372dc1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.261860 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fb79fcf30d485798c64c815e372dc1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.261931 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e977f7837e49fb93ef0fd18b619792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.262420 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e977f7837e49fb93ef0fd18b619792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.263036 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 3826594d300342c7a59e0eee989a873f (table default.Table_30f21f3fef874fdf94af3d3d043de7d7 [id=208684b1dfd044c6af3b7b4fa83b8c2b]) successfully deleted
I20240503 14:12:05.263381 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e977f7837e49fb93ef0fd18b619792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.263427 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet eb79edba14b646ef9be239b05a291d71 (table default.Table_3dd432e943764d9296fada2cf3c0ca47 [id=6791a22730fc49c59b2713384ee15850]) successfully deleted
W20240503 14:12:05.263703 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet eee989ff083041dc8dcd2cd1b220780d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee989ff083041dc8dcd2cd1b220780d already in progress: creating tablet
I20240503 14:12:05.267024 28661 ts_tablet_manager.cc:1888] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.267227 28661 log.cc:1196] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/b1de9717d0e44179b875174d5a76e4c3
I20240503 14:12:05.267760 28661 ts_tablet_manager.cc:1909] T b1de9717d0e44179b875174d5a76e4c3 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.270507 28604 tablet_bootstrap.cc:492] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:05.270741 28604 ts_tablet_manager.cc:1367] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.003s
I20240503 14:12:05.272137 28604 raft_consensus.cc:348] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.272464 28604 raft_consensus.cc:374] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.272576 28604 raft_consensus.cc:729] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.272920 28604 consensus_queue.cc:260] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.273564 28604 ts_tablet_manager.cc:1398] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:05.273952 28604 tablet_bootstrap.cc:492] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:05.276189 28604 tablet_bootstrap.cc:654] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.281250 28614 tablet_bootstrap.cc:492] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:05.281512 28614 ts_tablet_manager.cc:1367] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.224s user 0.008s sys 0.001s
I20240503 14:12:05.283193 28614 raft_consensus.cc:348] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.283635 28614 raft_consensus.cc:374] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.283797 28614 raft_consensus.cc:729] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.284279 28614 consensus_queue.cc:260] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.288048 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 90) took 2897 ms. Trace:
I20240503 14:12:05.288391 28661 rpcz_store.cc:276] 0503 14:12:02.390661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.390734 (+ 73us) service_pool.cc:225] Handling call
0503 14:12:05.258102 (+2867368us) ts_tablet_manager.cc:1120] Deleting tablet b1de9717d0e44179b875174d5a76e4c3
0503 14:12:05.258113 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.263269 (+ 5156us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.266828 (+ 3559us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.287912 (+ 21084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2867010}
I20240503 14:12:05.289304 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 91) took 2898 ms. Trace:
I20240503 14:12:05.290647 28604 tablet_bootstrap.cc:492] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:05.290793 28661 rpcz_store.cc:276] 0503 14:12:02.390792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.391140 (+ 348us) service_pool.cc:225] Handling call
0503 14:12:05.289006 (+2897866us) ts_tablet_manager.cc:1120] Deleting tablet a0e8b0eed57c4ad1bf6040507f6f8d4c
0503 14:12:05.289021 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.289237 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2897538}
I20240503 14:12:05.291150 28604 ts_tablet_manager.cc:1367] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20240503 14:12:05.291476 28661 tablet_replica.cc:330] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.291816 28661 raft_consensus.cc:2229] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.292112 28661 raft_consensus.cc:2258] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.292888 28661 ts_tablet_manager.cc:1875] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.294461 28604 raft_consensus.cc:348] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.295784 28604 raft_consensus.cc:374] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.296013 28604 raft_consensus.cc:729] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.296666 28604 consensus_queue.cc:260] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.296893 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet b1de9717d0e44179b875174d5a76e4c3 (table default.Table_466e18e3e6f346e798971d5b48a74866 [id=e2f2ebf7731a41e3ae451eddf555c454]) successfully deleted
W20240503 14:12:05.297277 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e8b0eed57c4ad1bf6040507f6f8d4c already in progress: creating tablet
I20240503 14:12:05.298611 28604 ts_tablet_manager.cc:1398] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:05.299288 28604 tablet_bootstrap.cc:492] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:05.300256 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bd5234791524eddb7edd493e39fb993 consensus state has no leader
I20240503 14:12:05.300773 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_666a0b3b2afb4a9fb91fb970e6c7a173" } modify_external_catalogs: true
I20240503 14:12:05.301051 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_666a0b3b2afb4a9fb91fb970e6c7a173" } modify_external_catalogs: true
I20240503 14:12:05.303085 28604 tablet_bootstrap.cc:654] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:05.305366 28984 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.222s user 0.000s sys 0.000s
I20240503 14:12:05.319412 28661 ts_tablet_manager.cc:1888] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.319725 28661 log.cc:1196] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/f9713865e09349ce9f9ae1d4297da2d0
I20240503 14:12:05.320322 28661 ts_tablet_manager.cc:1909] T f9713865e09349ce9f9ae1d4297da2d0 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.320791 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 92) took 2929 ms. Trace:
I20240503 14:12:05.320959 28661 rpcz_store.cc:276] 0503 14:12:02.390931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.391513 (+ 582us) service_pool.cc:225] Handling call
0503 14:12:05.291364 (+2899851us) ts_tablet_manager.cc:1120] Deleting tablet f9713865e09349ce9f9ae1d4297da2d0
0503 14:12:05.291374 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.314832 (+ 23458us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.319227 (+ 4395us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.320726 (+ 1499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899592}
I20240503 14:12:05.321626 28661 tablet_replica.cc:330] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.321920 28661 raft_consensus.cc:2229] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.322261 28661 raft_consensus.cc:2258] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.324613 28661 ts_tablet_manager.cc:1875] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.330982 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet f9713865e09349ce9f9ae1d4297da2d0 (table default.Table_36901c8428f346ae8117eb126334063b [id=15661374a8c648338c1386ad0b207148]) successfully deleted
I20240503 14:12:05.272182 28653 ts_tablet_manager.cc:1888] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.343489 28653 log.cc:1196] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/eb79edba14b646ef9be239b05a291d71
I20240503 14:12:05.287082 28600 tablet_bootstrap.cc:492] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:05.343945 28600 ts_tablet_manager.cc:1367] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.268s user 0.014s sys 0.032s
I20240503 14:12:05.345073 28653 ts_tablet_manager.cc:1909] T eb79edba14b646ef9be239b05a291d71 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:05.345149 28600 raft_consensus.cc:348] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.345512 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 103) took 2899 ms. Trace:
I20240503 14:12:05.345543 28600 raft_consensus.cc:374] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.345679 28600 raft_consensus.cc:729] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.346103 28600 consensus_queue.cc:260] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.346879 28600 ts_tablet_manager.cc:1398] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:05.347404 28600 tablet_bootstrap.cc:492] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:05.349907 28600 tablet_bootstrap.cc:654] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.350965 28661 ts_tablet_manager.cc:1888] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:12:05.352586 28661 log.cc:1196] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/5d637087c60144aaaa9b0965110d280e
I20240503 14:12:05.353209 28661 ts_tablet_manager.cc:1909] T 5d637087c60144aaaa9b0965110d280e P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.355818 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 93) took 2964 ms. Trace:
I20240503 14:12:05.356060 28661 rpcz_store.cc:276] 0503 14:12:02.391082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.425784 (+ 34702us) service_pool.cc:225] Handling call
0503 14:12:05.321446 (+2895662us) ts_tablet_manager.cc:1120] Deleting tablet 5d637087c60144aaaa9b0965110d280e
0503 14:12:05.321458 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.335303 (+ 13845us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.350783 (+ 15480us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.355734 (+ 4951us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2890741}
I20240503 14:12:05.356710 28661 tablet_replica.cc:330] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.357043 28661 raft_consensus.cc:2229] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:05.357579 28661 raft_consensus.cc:2258] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.357666 28600 tablet_bootstrap.cc:492] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:05.357944 28600 ts_tablet_manager.cc:1367] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:12:05.359081 28600 raft_consensus.cc:348] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.359400 28600 raft_consensus.cc:374] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.359513 28661 ts_tablet_manager.cc:1875] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.359583 28600 raft_consensus.cc:729] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.359916 28600 consensus_queue.cc:260] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.360605 28600 ts_tablet_manager.cc:1398] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:05.361029 28600 tablet_bootstrap.cc:492] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:05.363092 29037 log.cc:924] Time spent T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240503 14:12:05.363319 28600 tablet_bootstrap.cc:654] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.366695 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 5d637087c60144aaaa9b0965110d280e (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]) successfully deleted
I20240503 14:12:05.369184 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7c6f41bb9c5d40b0913264864ecc3528
I20240503 14:12:05.374277 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_42bdb5d25c3e4d778e5c09ceb819bec4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.375053 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42bdb5d25c3e4d778e5c09ceb819bec4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:05.376713 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b139ab0b5864024ac48215d46b0a7bd consensus state has no leader
I20240503 14:12:05.377300 28614 ts_tablet_manager.cc:1398] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.096s user 0.003s sys 0.000s
I20240503 14:12:05.377336 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a9eb6e575b54421b8692dd6fc829eaf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.378036 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9eb6e575b54421b8692dd6fc829eaf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.378172 28614 tablet_bootstrap.cc:492] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:05.379721 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 351d5bfb12c34c5f9de90e2df1b36fc1 consensus state has no leader
I20240503 14:12:05.380297 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3e4ec426791c4b11b43d3c36260eedd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.381013 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e4ec426791c4b11b43d3c36260eedd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:05.382848 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 52591.3us
I20240503 14:12:05.383203 28661 ts_tablet_manager.cc:1888] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.383445 28661 log.cc:1196] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/eb79edba14b646ef9be239b05a291d71
I20240503 14:12:05.383630 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8bfc56c4591c437da5b9686b726686b2
I20240503 14:12:05.384007 28661 ts_tablet_manager.cc:1909] T eb79edba14b646ef9be239b05a291d71 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.384366 28026 catalog_manager.cc:6554] ReplaceTablet: tablet 8bfc56c4591c437da5b9686b726686b2 deleted and replaced by tablet 7218cefb13ea4983b2111ed5fd0c997b
I20240503 14:12:05.384836 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_77748b64009a403a8a014fa47bff52b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.385303 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 94) took 2937 ms. Trace:
W20240503 14:12:05.385524 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77748b64009a403a8a014fa47bff52b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.385416 28661 rpcz_store.cc:276] 0503 14:12:02.448278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.457128 (+ 8850us) service_pool.cc:225] Handling call
0503 14:12:05.356618 (+2899490us) ts_tablet_manager.cc:1120] Deleting tablet eb79edba14b646ef9be239b05a291d71
0503 14:12:05.356629 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.363346 (+ 6717us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.367030 (+ 3684us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.385238 (+ 18208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2883981}
I20240503 14:12:05.386241 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 95) took 2768 ms. Trace:
I20240503 14:12:05.386365 28661 rpcz_store.cc:276] 0503 14:12:02.618063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.618127 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:05.385997 (+2767870us) ts_tablet_manager.cc:1120] Deleting tablet eee989ff083041dc8dcd2cd1b220780d
0503 14:12:05.386009 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.386147 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2767583}
I20240503 14:12:05.387491 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 105) took 2550 ms. Trace:
I20240503 14:12:05.387615 28661 rpcz_store.cc:276] 0503 14:12:02.837309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.837422 (+ 113us) service_pool.cc:225] Handling call
0503 14:12:05.387224 (+2549802us) ts_tablet_manager.cc:1120] Deleting tablet 59a91f22268346b38ae35f96b0d763f7
0503 14:12:05.387240 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.387395 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2548979}
I20240503 14:12:05.388304 28661 tablet_replica.cc:330] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.388566 28661 raft_consensus.cc:2229] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.388861 28661 raft_consensus.cc:2258] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.389602 28661 ts_tablet_manager.cc:1875] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.398221 28661 ts_tablet_manager.cc:1888] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.398562 28661 log.cc:1196] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7fe5d280a1654d80a2176c1def459e2e
I20240503 14:12:05.399269 28661 ts_tablet_manager.cc:1909] T 7fe5d280a1654d80a2176c1def459e2e P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.400341 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 106) took 2562 ms. Trace:
I20240503 14:12:05.400604 28661 rpcz_store.cc:276] 0503 14:12:02.837911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.837956 (+ 45us) service_pool.cc:225] Handling call
0503 14:12:05.388058 (+2550102us) ts_tablet_manager.cc:1120] Deleting tablet 7fe5d280a1654d80a2176c1def459e2e
0503 14:12:05.388069 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.394065 (+ 5996us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.398004 (+ 3939us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.400292 (+ 2288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2549731}
I20240503 14:12:05.401383 28661 tablet_replica.cc:330] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.401767 28661 raft_consensus.cc:2229] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.401767 28873 raft_consensus.cc:482] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.402053 28661 raft_consensus.cc:2258] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.401995 28873 raft_consensus.cc:504] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.403149 28661 ts_tablet_manager.cc:1875] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.403462 28873 leader_election.cc:288] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.403604 28984 tablet.cc:1717] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:05.403878 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbabe5beb9a42c69cb64e1b364d75a1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:05.404238 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbabe5beb9a42c69cb64e1b364d75a1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:05.404520 28492 raft_consensus.cc:2454] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
W20240503 14:12:05.404618 28150 leader_election.cc:341] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:05.405022 28150 leader_election.cc:302] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:05.405647 28873 raft_consensus.cc:2781] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.405885 28873 raft_consensus.cc:482] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.406085 28873 raft_consensus.cc:3037] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.406576 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.407682 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.407593 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.407747 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfc56c4591c437da5b9686b726686b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7218cefb13ea4983b2111ed5fd0c997b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.408102 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfc56c4591c437da5b9686b726686b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7218cefb13ea4983b2111ed5fd0c997b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.409039 28653 rpcz_store.cc:276] 0503 14:12:02.446090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.446231 (+ 141us) service_pool.cc:225] Handling call
0503 14:12:05.112411 (+2666180us) ts_tablet_manager.cc:1120] Deleting tablet eb79edba14b646ef9be239b05a291d71
0503 14:12:05.112429 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.208427 (+ 95998us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.271983 (+ 63556us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.345457 (+ 73474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2665884}
I20240503 14:12:05.409837 28653 tablet_replica.cc:330] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:05.410449 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet eb79edba14b646ef9be239b05a291d71 (table default.Table_3dd432e943764d9296fada2cf3c0ca47 [id=6791a22730fc49c59b2713384ee15850]) successfully deleted
I20240503 14:12:05.410697 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfc56c4591c437da5b9686b726686b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7218cefb13ea4983b2111ed5fd0c997b) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:05.411037 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet eee989ff083041dc8dcd2cd1b220780d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee989ff083041dc8dcd2cd1b220780d already in progress: creating tablet
I20240503 14:12:05.410951 28873 raft_consensus.cc:504] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:05.411427 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 59a91f22268346b38ae35f96b0d763f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a91f22268346b38ae35f96b0d763f7 already in progress: creating tablet
I20240503 14:12:05.411752 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7fe5d280a1654d80a2176c1def459e2e (table default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]) successfully deleted
I20240503 14:12:05.412256 28873 leader_election.cc:288] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.413280 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbabe5beb9a42c69cb64e1b364d75a1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:05.413985 28150 leader_election.cc:341] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:05.414261 28603 ts_tablet_manager.cc:1888] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.414503 28603 log.cc:1196] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7fe5d280a1654d80a2176c1def459e2e
I20240503 14:12:05.414889 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbabe5beb9a42c69cb64e1b364d75a1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:05.415349 28661 ts_tablet_manager.cc:1888] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.416119 28491 raft_consensus.cc:3037] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.416198 28661 log.cc:1196] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/4775ff1029e6449c83219030e3809984
I20240503 14:12:05.416838 28661 ts_tablet_manager.cc:1909] T 4775ff1029e6449c83219030e3809984 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.417861 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 107) took 2510 ms. Trace:
I20240503 14:12:05.418087 28661 rpcz_store.cc:276] 0503 14:12:02.907397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.907473 (+ 76us) service_pool.cc:225] Handling call
0503 14:12:05.401275 (+2493802us) ts_tablet_manager.cc:1120] Deleting tablet 4775ff1029e6449c83219030e3809984
0503 14:12:05.401289 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.408184 (+ 6895us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.415185 (+ 7001us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.417786 (+ 2601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2493522}
I20240503 14:12:05.418457 28653 raft_consensus.cc:2229] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:05.419132 28661 tablet_replica.cc:330] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.419431 28661 raft_consensus.cc:2229] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:05.420123 28661 raft_consensus.cc:2258] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.421242 28661 ts_tablet_manager.cc:1875] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.421810 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet eb79edba14b646ef9be239b05a291d71 (table default.Table_3dd432e943764d9296fada2cf3c0ca47 [id=6791a22730fc49c59b2713384ee15850]) successfully deleted
I20240503 14:12:05.422562 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 4775ff1029e6449c83219030e3809984 (table default.Table_2ad02256d653464e9528725b5e37c72c [id=36785e97f5c94a15bf717d499a33f558]) successfully deleted
I20240503 14:12:05.425247 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet fde0e59d911d4c1eb5125b1f869b396c
I20240503 14:12:05.426357 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3f8c7245a7a84b1e904cf130d5089dda" } new_table_name: "default.Table_f0de27b9aac24572a6433bbba3533f63" modify_external_catalogs: true
I20240503 14:12:05.430377 28653 raft_consensus.cc:2258] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.430718 28614 tablet_bootstrap.cc:654] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.419484 28491 raft_consensus.cc:2454] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:05.432775 28603 ts_tablet_manager.cc:1909] T 7fe5d280a1654d80a2176c1def459e2e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:05.432946 28150 leader_election.cc:302] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:05.433857 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 115) took 2580 ms. Trace:
I20240503 14:12:05.435438 28873 raft_consensus.cc:2781] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.435653 28873 raft_consensus.cc:686] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:05.435204 28980 tablet.cc:1722] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:05.436755 28984 tablet.cc:1722] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:05.438041 28661 ts_tablet_manager.cc:1888] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.438252 28661 log.cc:1196] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/bacf2d1091ee458c961c691093ea65a9
I20240503 14:12:05.438848 28661 ts_tablet_manager.cc:1909] T bacf2d1091ee458c961c691093ea65a9 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.439787 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 110) took 2464 ms. Trace:
I20240503 14:12:05.439954 28661 rpcz_store.cc:276] 0503 14:12:02.975569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.975721 (+ 152us) service_pool.cc:225] Handling call
0503 14:12:05.418995 (+2443274us) ts_tablet_manager.cc:1120] Deleting tablet bacf2d1091ee458c961c691093ea65a9
0503 14:12:05.419008 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.433904 (+ 14896us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.437847 (+ 3943us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.439730 (+ 1883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2442919}
I20240503 14:12:05.440688 28661 tablet_replica.cc:330] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.443320 28873 consensus_queue.cc:237] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.444468 28661 raft_consensus.cc:2229] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.444793 28661 raft_consensus.cc:2258] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.446782 28661 ts_tablet_manager.cc:1875] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.449143 28653 ts_tablet_manager.cc:1875] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.455391 28661 ts_tablet_manager.cc:1888] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.455798 28661 log.cc:1196] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ea3e9c030a424740bb2fb641328adfe1
I20240503 14:12:05.456393 28661 ts_tablet_manager.cc:1909] T ea3e9c030a424740bb2fb641328adfe1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.457430 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 108) took 2488 ms. Trace:
I20240503 14:12:05.457404 28603 rpcz_store.cc:276] 0503 14:12:02.853599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.855765 (+ 2166us) service_pool.cc:225] Handling call
0503 14:12:05.233488 (+2377723us) ts_tablet_manager.cc:1120] Deleting tablet 7fe5d280a1654d80a2176c1def459e2e
0503 14:12:05.233498 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.240198 (+ 6700us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.381463 (+141265us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.433753 (+ 52290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2308272}
I20240503 14:12:05.457657 28661 rpcz_store.cc:276] 0503 14:12:02.968894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.968999 (+ 105us) service_pool.cc:225] Handling call
0503 14:12:05.440571 (+2471572us) ts_tablet_manager.cc:1120] Deleting tablet ea3e9c030a424740bb2fb641328adfe1
0503 14:12:05.440583 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.451203 (+ 10620us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.455223 (+ 4020us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.457356 (+ 2133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3387,"tablet-delete.queue_time_us":2459857}
I20240503 14:12:05.458459 28661 tablet_replica.cc:330] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.458828 28661 raft_consensus.cc:2229] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.459167 28661 raft_consensus.cc:2258] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.459987 28661 ts_tablet_manager.cc:1875] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.466049 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 116) took 2541 ms. Trace:
I20240503 14:12:05.466475 28603 rpcz_store.cc:276] 0503 14:12:02.924551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.925077 (+ 526us) service_pool.cc:225] Handling call
0503 14:12:05.465645 (+2540568us) ts_tablet_manager.cc:1120] Deleting tablet 4775ff1029e6449c83219030e3809984
0503 14:12:05.465666 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.465865 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477296}
I20240503 14:12:05.467425 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 114) took 2613 ms. Trace:
I20240503 14:12:05.467694 28603 rpcz_store.cc:276] 0503 14:12:02.853435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.854199 (+ 764us) service_pool.cc:225] Handling call
0503 14:12:05.467136 (+2612937us) ts_tablet_manager.cc:1120] Deleting tablet 59a91f22268346b38ae35f96b0d763f7
0503 14:12:05.467149 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.467300 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2485231}
I20240503 14:12:05.468616 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 117) took 2501 ms. Trace:
I20240503 14:12:05.468744 28661 ts_tablet_manager.cc:1888] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.468909 28603 rpcz_store.cc:276] 0503 14:12:02.967474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.975211 (+ 7737us) service_pool.cc:225] Handling call
0503 14:12:05.468382 (+2493171us) ts_tablet_manager.cc:1120] Deleting tablet ea3e9c030a424740bb2fb641328adfe1
0503 14:12:05.468394 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.468557 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477751}
I20240503 14:12:05.469174 28661 log.cc:1196] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6106d52df3c54c2bb4abadd8605cd912
I20240503 14:12:05.469679 28603 tablet_replica.cc:330] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:05.469776 28661 ts_tablet_manager.cc:1909] T 6106d52df3c54c2bb4abadd8605cd912 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.470160 28603 raft_consensus.cc:2229] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.470502 28603 raft_consensus.cc:2258] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.470846 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 111) took 2494 ms. Trace:
I20240503 14:12:05.471081 28661 rpcz_store.cc:276] 0503 14:12:02.976636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.976887 (+ 251us) service_pool.cc:225] Handling call
0503 14:12:05.458295 (+2481408us) ts_tablet_manager.cc:1120] Deleting tablet 6106d52df3c54c2bb4abadd8605cd912
0503 14:12:05.458310 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.464294 (+ 5984us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.468518 (+ 4224us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.470790 (+ 2272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477361}
I20240503 14:12:05.471711 28661 tablet_replica.cc:330] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.472011 28661 raft_consensus.cc:2229] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.472198 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet fde0e59d911d4c1eb5125b1f869b396c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.473266 28661 raft_consensus.cc:2258] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.475052 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet fde0e59d911d4c1eb5125b1f869b396c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.475343 28661 ts_tablet_manager.cc:1875] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.475889 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7fe5d280a1654d80a2176c1def459e2e (table default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]) successfully deleted
W20240503 14:12:05.476308 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 4775ff1029e6449c83219030e3809984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4775ff1029e6449c83219030e3809984 already in progress: creating tablet
W20240503 14:12:05.476576 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 59a91f22268346b38ae35f96b0d763f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a91f22268346b38ae35f96b0d763f7 already in progress: creating tablet
W20240503 14:12:05.476981 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ea3e9c030a424740bb2fb641328adfe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea3e9c030a424740bb2fb641328adfe1 already in progress: creating tablet
I20240503 14:12:05.477248 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet bacf2d1091ee458c961c691093ea65a9 (table default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]) successfully deleted
I20240503 14:12:05.477490 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ea3e9c030a424740bb2fb641328adfe1 (table default.Table_cef464f155254da3a6e7a33965f98e2d [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
I20240503 14:12:05.477707 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6106d52df3c54c2bb4abadd8605cd912 (table default.Table_6270d949a6654e54b3cc252a54943112 [id=ada8758d8af14ac7bc31969934efb7d1]) successfully deleted
I20240503 14:12:05.482122 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 1044f34588d045ce898435b61a90503f from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:05.481906 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6fb64d577a8944049a29de7ed12a6255" } modify_external_catalogs: true
I20240503 14:12:05.483150 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7d8f2c48beb5472fb05f0cbd475f7182"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.483541 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6fb64d577a8944049a29de7ed12a6255" } modify_external_catalogs: true
W20240503 14:12:05.483932 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d8f2c48beb5472fb05f0cbd475f7182 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.485615 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_dbb3e0a033c84560a6458571881b25e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.486294 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbb3e0a033c84560a6458571881b25e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.488044 28951 tablet.cc:1722] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:05.495283 28661 ts_tablet_manager.cc:1888] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.495640 28661 log.cc:1196] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e85b6db1a21c4bca80c8bdec17e45c59
I20240503 14:12:05.496227 28661 ts_tablet_manager.cc:1909] T e85b6db1a21c4bca80c8bdec17e45c59 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.497252 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 112) took 2501 ms. Trace:
I20240503 14:12:05.497493 28661 rpcz_store.cc:276] 0503 14:12:02.995835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.998297 (+ 2462us) service_pool.cc:225] Handling call
0503 14:12:05.471614 (+2473317us) ts_tablet_manager.cc:1120] Deleting tablet e85b6db1a21c4bca80c8bdec17e45c59
0503 14:12:05.471625 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.491223 (+ 19598us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.495133 (+ 3910us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.497165 (+ 2032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2473008}
I20240503 14:12:05.498270 28661 tablet_replica.cc:330] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.498603 28661 raft_consensus.cc:2229] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.498914 28661 raft_consensus.cc:2258] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.499893 28661 ts_tablet_manager.cc:1875] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.519265 28600 tablet_bootstrap.cc:492] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:05.519681 28600 ts_tablet_manager.cc:1367] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.159s user 0.015s sys 0.038s
I20240503 14:12:05.521167 28600 raft_consensus.cc:348] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.521768 28600 raft_consensus.cc:374] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.521948 28600 raft_consensus.cc:729] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.475126 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet fde0e59d911d4c1eb5125b1f869b396c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.517493 28614 tablet_bootstrap.cc:492] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:05.522405 28600 consensus_queue.cc:260] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.523252 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e85b6db1a21c4bca80c8bdec17e45c59 (table default.Table_7afc535ef1bb440493a9e65dc3c86fde [id=52c4ea4489f24aa19d1671b4a5f88808]) successfully deleted
I20240503 14:12:05.523355 28600 ts_tablet_manager.cc:1398] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:05.527325 28600 tablet_bootstrap.cc:492] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:05.527948 28614 ts_tablet_manager.cc:1367] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.145s user 0.005s sys 0.000s
I20240503 14:12:05.530826 28600 tablet_bootstrap.cc:654] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.533026 28873 raft_consensus.cc:482] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.533416 28873 raft_consensus.cc:504] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.535374 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1693dd36a3d84d88a49690c537590083" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:05.536296 28150 leader_election.cc:341] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:05.536624 28873 leader_election.cc:288] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.538553 28614 raft_consensus.cc:348] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.539335 28614 raft_consensus.cc:374] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.539515 28614 raft_consensus.cc:729] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.539157 28603 ts_tablet_manager.cc:1875] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.542071 28614 consensus_queue.cc:260] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.544425 28614 ts_tablet_manager.cc:1398] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240503 14:12:05.552788 28614 tablet_bootstrap.cc:492] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:05.557293 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1693dd36a3d84d88a49690c537590083" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:05.558210 28150 leader_election.cc:341] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:05.558413 28150 leader_election.cc:302] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:05.559381 28661 ts_tablet_manager.cc:1888] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.559818 28661 log.cc:1196] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/9dbabe5beb9a42c69cb64e1b364d75a1
I20240503 14:12:05.559756 28873 raft_consensus.cc:2726] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:05.561172 28661 ts_tablet_manager.cc:1909] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.562238 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 113) took 2560 ms. Trace:
I20240503 14:12:05.562517 28661 rpcz_store.cc:276] 0503 14:12:03.002050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.004922 (+ 2872us) service_pool.cc:225] Handling call
0503 14:12:05.498128 (+2493206us) ts_tablet_manager.cc:1120] Deleting tablet 9dbabe5beb9a42c69cb64e1b364d75a1
0503 14:12:05.498140 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.511613 (+ 13473us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.559160 (+ 47547us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.562180 (+ 3020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2492805}
I20240503 14:12:05.563302 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 124) took 2398 ms. Trace:
I20240503 14:12:05.563566 28661 rpcz_store.cc:276] 0503 14:12:03.164757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.164841 (+ 84us) service_pool.cc:225] Handling call
0503 14:12:05.563036 (+2398195us) ts_tablet_manager.cc:1120] Deleting tablet 40da5f5fff94462d8af740f3ee337e76
0503 14:12:05.563049 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.563234 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397727}
I20240503 14:12:05.564339 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 125) took 2213 ms. Trace:
I20240503 14:12:05.564592 28661 rpcz_store.cc:276] 0503 14:12:03.351147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.351215 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:05.564073 (+2212858us) ts_tablet_manager.cc:1120] Deleting tablet cae4969fc9474fdbaeae1b4d09e4875b
0503 14:12:05.564118 (+ 45us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.564260 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2212599}
I20240503 14:12:05.565124 28661 tablet_replica.cc:330] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.565296 28614 tablet_bootstrap.cc:654] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.565502 28661 raft_consensus.cc:2229] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:05.565622 28492 raft_consensus.cc:3037] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.565837 28661 pending_rounds.cc:62] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:05.565970 28661 pending_rounds.cc:68] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597667955101696 op_type: NO_OP noop_request { }
I20240503 14:12:05.566152 28661 raft_consensus.cc:2866] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:05.566370 28661 raft_consensus.cc:2258] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.567179 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 9dbabe5beb9a42c69cb64e1b364d75a1 (table default.Table_6d7feb39cce84e66b75914a6bcbcfcc9 [id=56b86fbb2fff40cfa3729f88092d57ca]) successfully deleted
W20240503 14:12:05.567678 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 40da5f5fff94462d8af740f3ee337e76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40da5f5fff94462d8af740f3ee337e76 already in progress: creating tablet
W20240503 14:12:05.568048 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet cae4969fc9474fdbaeae1b4d09e4875b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cae4969fc9474fdbaeae1b4d09e4875b already in progress: creating tablet
I20240503 14:12:05.536612 28653 ts_tablet_manager.cc:1888] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.571023 28653 log.cc:1196] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/b1de9717d0e44179b875174d5a76e4c3
W20240503 14:12:05.572203 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f1aa939aced4556ae0f0de0e62d1f97 consensus state has no leader
I20240503 14:12:05.572854 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9d9d27627b064475a2f86b5ddcf4236f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.573361 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 240abf8c2b2343408fc98a8dac1fdf09 (DEFAULT_TABLE table=default.Table_bbc0f1621f4842c2a8b5913acd913703 [id=c2e9f50f96514774acb83f844cdd8692]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:05.573626 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d9d27627b064475a2f86b5ddcf4236f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.573920 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 0926e523ef454a2089f4d40a328c1c6d (DEFAULT_TABLE table=default.Table_5e7ed1c5e8934a81bfeba19fe7452080 [id=3e3945bf21d24fedb9b811e4e3648567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.573920 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240abf8c2b2343408fc98a8dac1fdf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.574769 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 240abf8c2b2343408fc98a8dac1fdf09 (DEFAULT_TABLE table=default.Table_bbc0f1621f4842c2a8b5913acd913703 [id=c2e9f50f96514774acb83f844cdd8692]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.575249 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240abf8c2b2343408fc98a8dac1fdf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.576324 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0926e523ef454a2089f4d40a328c1c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.576994 28029 catalog_manager.cc:5484] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:05.577636 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 2 got 0
I20240503 14:12:05.580066 28319 tablet_service.cc:1467] Processing CreateTablet for tablet e135b60660f448a7974520f2f87cf93a (DEFAULT_TABLE table=default.Table_00e404138f154498b730db1de48ded05 [id=6edae5b6460546ae80b1f25ffec68d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.580675 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e135b60660f448a7974520f2f87cf93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.580768 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 240abf8c2b2343408fc98a8dac1fdf09 (DEFAULT_TABLE table=default.Table_bbc0f1621f4842c2a8b5913acd913703 [id=c2e9f50f96514774acb83f844cdd8692]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.581306 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240abf8c2b2343408fc98a8dac1fdf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.584602 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 0926e523ef454a2089f4d40a328c1c6d (DEFAULT_TABLE table=default.Table_5e7ed1c5e8934a81bfeba19fe7452080 [id=3e3945bf21d24fedb9b811e4e3648567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.584602 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 0926e523ef454a2089f4d40a328c1c6d (DEFAULT_TABLE table=default.Table_5e7ed1c5e8934a81bfeba19fe7452080 [id=3e3945bf21d24fedb9b811e4e3648567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.585161 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0926e523ef454a2089f4d40a328c1c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.585225 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0926e523ef454a2089f4d40a328c1c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.588151 28603 ts_tablet_manager.cc:1888] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:05.588445 28603 log.cc:1196] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/2502a13e8cbc4a3abfed225f94c44d2f
I20240503 14:12:05.589018 28603 ts_tablet_manager.cc:1909] T 2502a13e8cbc4a3abfed225f94c44d2f P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:05.590132 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 118) took 2605 ms. Trace:
I20240503 14:12:05.590337 28603 rpcz_store.cc:276] 0503 14:12:02.984332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.984511 (+ 179us) service_pool.cc:225] Handling call
0503 14:12:05.469564 (+2485053us) ts_tablet_manager.cc:1120] Deleting tablet 2502a13e8cbc4a3abfed225f94c44d2f
0503 14:12:05.469577 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.563249 (+ 93672us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.587952 (+ 24703us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.590032 (+ 2080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2454706}
I20240503 14:12:05.590871 28603 tablet_replica.cc:330] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:05.592219 28661 ts_tablet_manager.cc:1875] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.596371 28604 tablet_bootstrap.cc:492] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:05.596762 28604 ts_tablet_manager.cc:1367] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.298s user 0.033s sys 0.023s
I20240503 14:12:05.597919 28604 raft_consensus.cc:348] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.598598 28604 raft_consensus.cc:374] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.591143 28603 raft_consensus.cc:2229] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.602064 28603 raft_consensus.cc:2258] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.603696 28604 raft_consensus.cc:729] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.607095 28614 tablet_bootstrap.cc:492] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:05.607427 28614 ts_tablet_manager.cc:1367] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20240503 14:12:05.609002 28184 tablet_service.cc:1467] Processing CreateTablet for tablet e135b60660f448a7974520f2f87cf93a (DEFAULT_TABLE table=default.Table_00e404138f154498b730db1de48ded05 [id=6edae5b6460546ae80b1f25ffec68d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.609184 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 (DEFAULT_TABLE table=default.Table_852b77a0ef2e4ee6bda7be721348aff8 [id=0a16a4e85e1842f1a2880d2873d363b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.609520 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e135b60660f448a7974520f2f87cf93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.609474 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 2502a13e8cbc4a3abfed225f94c44d2f (table default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:05.609628 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ba571a81eb4240a4a4a4a08cb879c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.611891 28653 ts_tablet_manager.cc:1909] T b1de9717d0e44179b875174d5a76e4c3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:05.612804 28026 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet f59e5eb449644d82a66056c06d1b837d
I20240503 14:12:05.613304 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b0e228f901e4427ca9bbfc14b02c1f7f" } modify_external_catalogs: true
I20240503 14:12:05.613570 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b0e228f901e4427ca9bbfc14b02c1f7f" } modify_external_catalogs: true
I20240503 14:12:05.615643 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c355201f2d654091bbf81fd66561a942"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.616371 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c355201f2d654091bbf81fd66561a942 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.616626 28614 raft_consensus.cc:348] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.616987 28614 raft_consensus.cc:374] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.616986 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 98) took 3219 ms. Trace:
I20240503 14:12:05.617094 28653 rpcz_store.cc:276] 0503 14:12:02.397908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.499837 (+101929us) service_pool.cc:225] Handling call
0503 14:12:05.409722 (+2909885us) ts_tablet_manager.cc:1120] Deleting tablet b1de9717d0e44179b875174d5a76e4c3
0503 14:12:05.409736 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.430032 (+ 20296us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x616000112054. stack: 00007f2f23940756 00007f2f23940510 00007f2f2297348d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2f1aea00 00007f2f30488b83 00007f2f32f1c57c 00007f2f32f4e8d4 00007f2f32f2fbc0 00007f2f32f2f94d 0000000000a60bcc 00007f2f23986b74 00007f2f2398a19b 00007f2f2398a018
0503 14:12:05.519685 (+ 89653us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.536443 (+ 16758us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.616914 (+ 80471us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19016448,"tablet-delete.queue_time_us":2903708}
I20240503 14:12:05.618165 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet b1de9717d0e44179b875174d5a76e4c3 (table default.Table_466e18e3e6f346e798971d5b48a74866 [id=e2f2ebf7731a41e3ae451eddf555c454]) successfully deleted
I20240503 14:12:05.618575 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_106726a25eb74a22951f48bee14b67dd" } new_table_name: "default.Table_e5219472726e4702be79f785858e5ded" modify_external_catalogs: true
I20240503 14:12:05.619305 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 104) took 3002 ms. Trace:
I20240503 14:12:05.620046 28614 raft_consensus.cc:729] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.620179 28653 rpcz_store.cc:276] 0503 14:12:02.617183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.617246 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:05.618577 (+3001331us) ts_tablet_manager.cc:1120] Deleting tablet eee989ff083041dc8dcd2cd1b220780d
0503 14:12:05.618590 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.619227 (+ 637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3000989}
I20240503 14:12:05.620565 28614 consensus_queue.cc:260] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:05.620869 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet eee989ff083041dc8dcd2cd1b220780d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee989ff083041dc8dcd2cd1b220780d already in progress: creating tablet
I20240503 14:12:05.621430 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 (DEFAULT_TABLE table=default.Table_852b77a0ef2e4ee6bda7be721348aff8 [id=0a16a4e85e1842f1a2880d2873d363b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.623055 28603 ts_tablet_manager.cc:1875] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.623414 28614 ts_tablet_manager.cc:1398] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240503 14:12:05.624357 28614 tablet_bootstrap.cc:492] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:05.626047 28460 tablet_service.cc:1467] Processing CreateTablet for tablet e135b60660f448a7974520f2f87cf93a (DEFAULT_TABLE table=default.Table_00e404138f154498b730db1de48ded05 [id=6edae5b6460546ae80b1f25ffec68d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.626590 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ba571a81eb4240a4a4a4a08cb879c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.629343 28540 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:05.632742 28614 tablet_bootstrap.cc:654] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.634168 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e135b60660f448a7974520f2f87cf93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.619794 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 (DEFAULT_TABLE table=default.Table_852b77a0ef2e4ee6bda7be721348aff8 [id=0a16a4e85e1842f1a2880d2873d363b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.636708 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ba571a81eb4240a4a4a4a08cb879c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.636811 28604 consensus_queue.cc:260] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.637673 28604 ts_tablet_manager.cc:1398] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240503 14:12:05.639333 28603 ts_tablet_manager.cc:1888] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.640672 28603 log.cc:1196] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/bacf2d1091ee458c961c691093ea65a9
I20240503 14:12:05.642565 28604 tablet_bootstrap.cc:492] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:05.642900 28603 ts_tablet_manager.cc:1909] T bacf2d1091ee458c961c691093ea65a9 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:05.644199 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 120) took 2653 ms. Trace:
I20240503 14:12:05.644800 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 115) took 2807 ms. Trace:
I20240503 14:12:05.645026 28653 rpcz_store.cc:276] 0503 14:12:02.837242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.837295 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:05.644492 (+2807197us) ts_tablet_manager.cc:1120] Deleting tablet 59a91f22268346b38ae35f96b0d763f7
0503 14:12:05.644509 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.644699 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2805455}
W20240503 14:12:05.645906 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 59a91f22268346b38ae35f96b0d763f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a91f22268346b38ae35f96b0d763f7 already in progress: creating tablet
I20240503 14:12:05.646582 28653 tablet_replica.cc:330] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:05.646955 28653 raft_consensus.cc:2229] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.647447 28653 raft_consensus.cc:2258] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.648720 28614 tablet_bootstrap.cc:492] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:05.649654 28614 ts_tablet_manager.cc:1367] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20240503 14:12:05.650897 28614 raft_consensus.cc:348] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.651355 28614 raft_consensus.cc:374] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.651515 28614 raft_consensus.cc:729] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.651927 28614 consensus_queue.cc:260] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.652428 28604 tablet_bootstrap.cc:654] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.653861 28403 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:05.653908 28614 ts_tablet_manager.cc:1398] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:05.654378 28614 tablet_bootstrap.cc:492] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:05.648927 28603 rpcz_store.cc:276] 0503 14:12:02.990501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.990679 (+ 178us) service_pool.cc:225] Handling call
0503 14:12:05.590710 (+2600031us) ts_tablet_manager.cc:1120] Deleting tablet bacf2d1091ee458c961c691093ea65a9
0503 14:12:05.590724 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.634989 (+ 44265us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.639142 (+ 4153us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.644144 (+ 5002us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2575209}
I20240503 14:12:05.655383 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1044f34588d045ce898435b61a90503f
I20240503 14:12:05.656106 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 1044f34588d045ce898435b61a90503f deleted and replaced by tablet daf5d153ab5142c0902cc6e96629f412
I20240503 14:12:05.656441 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet a8d2600031df4d77b9afe636a9b5ad40 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:05.657053 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 559342d357d64f9eb9e3ab42ccb75942 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:05.656530 28754 raft_consensus.cc:482] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.657275 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044f34588d045ce898435b61a90503f with delete_type TABLET_DATA_DELETED (replaced by tablet daf5d153ab5142c0902cc6e96629f412) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.657840 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1693dd36a3d84d88a49690c537590083
I20240503 14:12:05.657752 28754 raft_consensus.cc:504] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.658047 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet bacf2d1091ee458c961c691093ea65a9 (table default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]) successfully deleted
I20240503 14:12:05.658408 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 1693dd36a3d84d88a49690c537590083 deleted and replaced by tablet 6c7c5dfac02d4dfe932996d2a37fe031
I20240503 14:12:05.658610 28032 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 234) took 1211 ms. Trace:
I20240503 14:12:05.658648 28754 leader_election.cc:288] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.658897 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1693dd36a3d84d88a49690c537590083 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7c5dfac02d4dfe932996d2a37fe031) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.658952 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044f34588d045ce898435b61a90503f with delete_type TABLET_DATA_DELETED (replaced by tablet daf5d153ab5142c0902cc6e96629f412) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.658731 28032 rpcz_store.cc:276] 0503 14:12:04.447515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.079809 (+632294us) service_pool.cc:225] Handling call
0503 14:12:05.658516 (+578707us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:05.599379 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:05.599477 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:05.599483 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:05.599485 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:05.599610 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:05.599622 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:05.599627 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:05.599701 (+ 74us) tablet.cc:699] Row locks acquired
0503 14:12:05.599704 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:05.599740 (+ 36us) write_op.cc:270] Start()
0503 14:12:05.599770 (+ 30us) write_op.cc:276] Timestamp: P: 1714745525599715 usec, L: 0
0503 14:12:05.599773 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:05.599918 (+ 145us) log.cc:842] Serialized 628 byte log entry
0503 14:12:05.651930 (+ 52012us) op_driver.cc:444] REPLICATION: finished
0503 14:12:05.653931 (+ 2001us) write_op.cc:301] APPLY: starting
0503 14:12:05.653969 (+ 38us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:05.654030 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:05.654067 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:05.654182 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:05.654318 (+ 136us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:05.654325 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:05.654402 (+ 77us) log.cc:842] Serialized 29 byte log entry
0503 14:12:05.654457 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.654475 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:05.654482 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:05.654544 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.654548 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1972,"apply.run_cpu_time_us":669,"apply.run_wall_time_us":667,"mutex_wait_us":17313,"num_ops":2,"prepare.queue_time_us":21354,"prepare.run_cpu_time_us":590,"prepare.run_wall_time_us":589,"replication_time_us":52144}]]}
I20240503 14:12:05.659797 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet 1693dd36a3d84d88a49690c537590083 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7c5dfac02d4dfe932996d2a37fe031) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.659754 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0c010da29c3645e6a22acf46e3a01775" } new_table_name: "default.Table_17677d89d0664bb4b835cd1c24a9f27c" modify_external_catalogs: true
I20240503 14:12:05.661083 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7d57fc4486457e9848daad79f04356" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:05.661468 28212 raft_consensus.cc:2454] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:05.661639 28653 ts_tablet_manager.cc:1875] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.661657 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_852b77a0ef2e4ee6bda7be721348aff8" } modify_external_catalogs: true
I20240503 14:12:05.661908 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_852b77a0ef2e4ee6bda7be721348aff8" } modify_external_catalogs: true
I20240503 14:12:05.662531 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7d57fc4486457e9848daad79f04356" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:05.662850 28356 raft_consensus.cc:2454] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:05.663252 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 121) took 2647 ms. Trace:
I20240503 14:12:05.663417 28427 leader_election.cc:302] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:05.663419 28603 rpcz_store.cc:276] 0503 14:12:03.016355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.017238 (+ 883us) service_pool.cc:225] Handling call
0503 14:12:05.655611 (+2638373us) ts_tablet_manager.cc:1120] Deleting tablet 9dbabe5beb9a42c69cb64e1b364d75a1
0503 14:12:05.661322 (+ 5711us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.663184 (+ 1862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2638099}
I20240503 14:12:05.663801 28754 raft_consensus.cc:2781] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.663931 28754 raft_consensus.cc:482] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.664019 28603 tablet_replica.cc:330] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:05.664074 28754 raft_consensus.cc:3037] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.665001 28028 catalog_manager.cc:5484] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:05.665585 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 23186a7a0dfb432e813c8a05e97e46dd (table default.Table_afc737ad630d4de6b239f807771fa6a1 [id=68a4a475afc24b95b5388e43deb4632b]). Expected version 1 got 0
I20240503 14:12:05.665771 28603 raft_consensus.cc:2229] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:05.665961 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]). Expected version 1 got 0
I20240503 14:12:05.666218 28603 raft_consensus.cc:2258] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.667291 28603 ts_tablet_manager.cc:1875] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.667476 28754 raft_consensus.cc:504] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.667964 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 129717ef137c41c5a08d401c64164187
I20240503 14:12:05.668476 28754 leader_election.cc:288] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.668670 28025 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 237) took 1119 ms. Trace:
I20240503 14:12:05.668659 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 1693dd36a3d84d88a49690c537590083 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7c5dfac02d4dfe932996d2a37fe031) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.667733 28614 tablet_bootstrap.cc:654] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.668828 28025 rpcz_store.cc:276] 0503 14:12:04.548826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.091169 (+542343us) service_pool.cc:225] Handling call
0503 14:12:05.091856 (+ 687us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:05.092019 (+ 163us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:05.099271 (+ 7252us) catalog_manager.cc:2758] Locking tablets
0503 14:12:05.576160 (+476889us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:05.657316 (+ 81156us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:05.657427 (+ 111us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:05.657484 (+ 57us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:05.668596 (+ 11112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:05.598495 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:05.598633 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:05.598641 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:05.598643 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:05.598771 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:05.598785 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:05.598790 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:05.598994 (+ 204us) tablet.cc:699] Row locks acquired
0503 14:12:05.598997 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:05.599043 (+ 46us) write_op.cc:270] Start()
0503 14:12:05.599079 (+ 36us) write_op.cc:276] Timestamp: P: 1714745525599015 usec, L: 0
0503 14:12:05.599083 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:05.599280 (+ 197us) log.cc:842] Serialized 731 byte log entry
0503 14:12:05.651892 (+ 52612us) op_driver.cc:444] REPLICATION: finished
0503 14:12:05.653437 (+ 1545us) write_op.cc:301] APPLY: starting
0503 14:12:05.653448 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:05.653488 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:05.653528 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:05.653580 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:05.653619 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:05.653627 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:05.653686 (+ 59us) log.cc:842] Serialized 29 byte log entry
0503 14:12:05.653738 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.653775 (+ 37us) write_op.cc:454] Released schema lock
0503 14:12:05.653782 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:05.653839 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:05.653843 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10344,"child_traces":[["op",{"apply.queue_time_us":1515,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":454,"mutex_wait_us":18267,"num_ops":2,"prepare.queue_time_us":21463,"prepare.run_cpu_time_us":847,"prepare.run_wall_time_us":846,"replication_time_us":52796}]]}
I20240503 14:12:05.669570 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7d57fc4486457e9848daad79f04356" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:05.669836 28212 raft_consensus.cc:3037] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.671919 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044f34588d045ce898435b61a90503f with delete_type TABLET_DATA_DELETED (replaced by tablet daf5d153ab5142c0902cc6e96629f412) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.673070 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cca3611a96024dd49da0111bbdf07abd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.673831 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cca3611a96024dd49da0111bbdf07abd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.675053 28212 raft_consensus.cc:2454] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:05.677322 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7d57fc4486457e9848daad79f04356" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:05.677637 28356 raft_consensus.cc:3037] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.677878 28427 leader_election.cc:302] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:05.679124 28754 raft_consensus.cc:2781] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.679385 28754 raft_consensus.cc:686] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:05.679718 28754 consensus_queue.cc:237] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.682050 28356 raft_consensus.cc:2454] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:05.685148 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 129717ef137c41c5a08d401c64164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.685251 28614 tablet_bootstrap.cc:492] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:05.685606 28614 ts_tablet_manager.cc:1367] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20240503 14:12:05.685961 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 129717ef137c41c5a08d401c64164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:05.686547 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9dbabe5beb9a42c69cb64e1b364d75a1 already in progress: creating tablet
I20240503 14:12:05.686969 28614 raft_consensus.cc:348] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.687340 28614 raft_consensus.cc:374] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.687508 28614 raft_consensus.cc:729] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.687896 28614 consensus_queue.cc:260] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.690164 28614 ts_tablet_manager.cc:1398] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:05.690639 28614 tablet_bootstrap.cc:492] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:05.697209 28873 raft_consensus.cc:482] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.697518 28873 raft_consensus.cc:504] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.698405 28873 leader_election.cc:288] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:05.699667 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d2600031df4d77b9afe636a9b5ad40" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:05.701359 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d2600031df4d77b9afe636a9b5ad40" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:05.702040 28600 tablet_bootstrap.cc:492] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:05.702783 28600 ts_tablet_manager.cc:1367] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.175s user 0.017s sys 0.026s
I20240503 14:12:05.703964 28600 raft_consensus.cc:348] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:05.704632 28150 leader_election.cc:341] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:05.704979 28150 leader_election.cc:341] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:05.707217 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f51834695b8b4a729a088da81eee2e47
I20240503 14:12:05.707129 28614 tablet_bootstrap.cc:654] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.707343 28603 ts_tablet_manager.cc:1888] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.708510 28603 log.cc:1196] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/92c800e997e944158f7c839dd0e00d44
I20240503 14:12:05.708364 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e0522da637fa4e2e880f9e9c076d3505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.708843 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet f51834695b8b4a729a088da81eee2e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.709077 28603 ts_tablet_manager.cc:1909] T 92c800e997e944158f7c839dd0e00d44 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:05.709225 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0522da637fa4e2e880f9e9c076d3505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.709906 28150 leader_election.cc:302] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:05.710279 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet f51834695b8b4a729a088da81eee2e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.710574 28873 raft_consensus.cc:2726] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:05.710813 28600 raft_consensus.cc:374] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.710956 28600 raft_consensus.cc:729] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.711241 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_21a60d0d25074a54abc69f6309875813" } new_table_name: "default.Table_d57cf3dd2d394751bf1d54730635a8dd" modify_external_catalogs: true
I20240503 14:12:05.711387 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 119) took 2721 ms. Trace:
I20240503 14:12:05.711535 28603 rpcz_store.cc:276] 0503 14:12:02.990314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.991251 (+ 937us) service_pool.cc:225] Handling call
0503 14:12:05.663911 (+2672660us) ts_tablet_manager.cc:1120] Deleting tablet 92c800e997e944158f7c839dd0e00d44
0503 14:12:05.663924 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.675388 (+ 11464us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.694553 (+ 19165us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.711259 (+ 16706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2639305}
I20240503 14:12:05.712210 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 132) took 2586 ms. Trace:
I20240503 14:12:05.712280 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 92c800e997e944158f7c839dd0e00d44 (table default.Table_5f10f98658754315a6cc9b1d7ffc19cb [id=d9b865dc744148df9fc1ab6e79c5043d]) successfully deleted
I20240503 14:12:05.712334 28603 rpcz_store.cc:276] 0503 14:12:03.125293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.125530 (+ 237us) service_pool.cc:225] Handling call
0503 14:12:05.711913 (+2586383us) ts_tablet_manager.cc:1120] Deleting tablet 40da5f5fff94462d8af740f3ee337e76
0503 14:12:05.711923 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.712111 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2568007}
I20240503 14:12:05.712895 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 133) took 2089 ms. Trace:
W20240503 14:12:05.712913 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 40da5f5fff94462d8af740f3ee337e76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40da5f5fff94462d8af740f3ee337e76 already in progress: creating tablet
W20240503 14:12:05.713037 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2b861dccd964092ae0f06668c9b58fe consensus state has no leader
I20240503 14:12:05.713016 28603 rpcz_store.cc:276] 0503 14:12:03.623171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.630349 (+ 7178us) service_pool.cc:225] Handling call
0503 14:12:05.712630 (+2082281us) ts_tablet_manager.cc:1120] Deleting tablet cae4969fc9474fdbaeae1b4d09e4875b
0503 14:12:05.712640 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.712762 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2081979}
I20240503 14:12:05.713536 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 148) took 2021 ms. Trace:
I20240503 14:12:05.713526 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_642763ad4fae43e2b4f0d0c764202168" } new_table_name: "default.Table_355c3b14ae4f480dba47c2c90dc9dba2" modify_external_catalogs: true
I20240503 14:12:05.713651 28603 rpcz_store.cc:276] 0503 14:12:03.692262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.692392 (+ 130us) service_pool.cc:225] Handling call
0503 14:12:05.713301 (+2020909us) ts_tablet_manager.cc:1120] Deleting tablet a8d2600031df4d77b9afe636a9b5ad40
0503 14:12:05.713311 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.713443 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2016219}
W20240503 14:12:05.713553 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet cae4969fc9474fdbaeae1b4d09e4875b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cae4969fc9474fdbaeae1b4d09e4875b already in progress: creating tablet
I20240503 14:12:05.714036 28603 tablet_replica.cc:330] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:05.714277 28603 raft_consensus.cc:2229] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:05.714315 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a8d2600031df4d77b9afe636a9b5ad40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8d2600031df4d77b9afe636a9b5ad40 already in progress: creating tablet
I20240503 14:12:05.714730 28603 raft_consensus.cc:2258] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.715133 28600 consensus_queue.cc:260] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.718209 28263 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:05.711396 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 129717ef137c41c5a08d401c64164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.725041 28600 ts_tablet_manager.cc:1398] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 14:12:05.731407 28653 ts_tablet_manager.cc:1888] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.731585 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 47ba571a81eb4240a4a4a4a08cb879c5
W20240503 14:12:05.735937 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65853212c2404597b015ddc8ea99ee47 has no consensus state
I20240503 14:12:05.736416 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 37dc7d96357743089372baf71654b1e3 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:05.737183 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5eec9210201643c3a011af892345d421" } modify_external_catalogs: true
I20240503 14:12:05.737473 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5eec9210201643c3a011af892345d421" } modify_external_catalogs: true
I20240503 14:12:05.740840 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.743049 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c6f1f5b922f549679f50474fd6412cfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.737557 28600 tablet_bootstrap.cc:492] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:05.741883 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:05.743908 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6f1f5b922f549679f50474fd6412cfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.744005 28653 log.cc:1196] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7fe5d280a1654d80a2176c1def459e2e
I20240503 14:12:05.744268 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.744630 28653 ts_tablet_manager.cc:1909] T 7fe5d280a1654d80a2176c1def459e2e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:05.745637 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 116) took 2903 ms. Trace:
I20240503 14:12:05.746666 29027 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0690a516c89446d85c88cdf64fa4238. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:05.747063 29027 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:05.747548 28603 ts_tablet_manager.cc:1875] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.749581 28102 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:05.749905 28102 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:05.750816 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_bc583d61a09e4e87b183960e67bbc8b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.751785 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc583d61a09e4e87b183960e67bbc8b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:05.752657 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.746088 28600 tablet_bootstrap.cc:654] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.764837 28653 rpcz_store.cc:276] 0503 14:12:02.842169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.842234 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:05.646425 (+2804191us) ts_tablet_manager.cc:1120] Deleting tablet 7fe5d280a1654d80a2176c1def459e2e
0503 14:12:05.646438 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.715327 (+ 68889us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.731195 (+ 15868us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.745582 (+ 14387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2792176}
I20240503 14:12:05.765658 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7fe5d280a1654d80a2176c1def459e2e (table default.Table_73db66f21afd468d887601ce8148489a [id=8b305f063c044901b54e74d473d73cb8]) successfully deleted
I20240503 14:12:05.766886 28653 tablet_replica.cc:330] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:05.767210 28653 raft_consensus.cc:2229] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.767450 28653 raft_consensus.cc:2258] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.768252 28653 ts_tablet_manager.cc:1875] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.773159 29078 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:12:05.773910 29078 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:12:05.774605 28614 tablet_bootstrap.cc:492] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:05.774907 28614 ts_tablet_manager.cc:1367] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.014s
I20240503 14:12:05.775951 28614 raft_consensus.cc:348] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.777210 28614 raft_consensus.cc:374] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.777846 28614 raft_consensus.cc:729] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.778959 28614 consensus_queue.cc:260] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.782480 28614 ts_tablet_manager.cc:1398] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:05.786798 28600 tablet_bootstrap.cc:492] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:05.787147 28600 ts_tablet_manager.cc:1367] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.008s
I20240503 14:12:05.788430 28600 raft_consensus.cc:348] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.788892 28600 raft_consensus.cc:374] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.789147 28600 raft_consensus.cc:729] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.789222 28603 ts_tablet_manager.cc:1888] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:05.789584 28600 consensus_queue.cc:260] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.798619 28600 ts_tablet_manager.cc:1398] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:12:05.805495 28600 tablet_bootstrap.cc:492] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:05.809518 28600 tablet_bootstrap.cc:654] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.810694 29078 leader_election.cc:288] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:12:05.799103 28614 tablet_bootstrap.cc:492] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:05.823405 28603 log.cc:1196] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a2bfdcb8e2b241ca9a6922fd8491e27b
I20240503 14:12:05.825181 28600 tablet_bootstrap.cc:492] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:05.825491 28600 ts_tablet_manager.cc:1367] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20240503 14:12:05.827560 28600 raft_consensus.cc:348] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.827950 28600 raft_consensus.cc:374] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.828173 28600 raft_consensus.cc:729] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.828720 28600 consensus_queue.cc:260] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:05.832758 28186 tablet_service.cc:1467] Processing CreateTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 (DEFAULT_TABLE table=default.Table_3e4ec426791c4b11b43d3c36260eedd4 [id=c14f7d47ddd64652aa553945f775aec6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:05.832904 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37dc7d96357743089372baf71654b1e3 consensus state has no leader
I20240503 14:12:05.833328 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25c852aa7aa4fbfafdacec43f5bc426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.833465 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c7035b3f783948498dfb7bd5ed14aad8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.834170 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7035b3f783948498dfb7bd5ed14aad8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:05.835803 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6371e68ee85a47cdb0e8e21f7124d94a consensus state has no leader
I20240503 14:12:05.836274 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6d9dd77419724e2a9b5b1efeea5eb053" } new_table_name: "default.Table_aa957a5e7a874114aaaec61023360b59" modify_external_catalogs: true
I20240503 14:12:05.836624 28600 ts_tablet_manager.cc:1398] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20240503 14:12:05.838276 28614 tablet_bootstrap.cc:654] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.838429 28324 tablet_service.cc:1467] Processing CreateTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 (DEFAULT_TABLE table=default.Table_3e4ec426791c4b11b43d3c36260eedd4 [id=c14f7d47ddd64652aa553945f775aec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.838321 28031 catalog_manager.cc:5484] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:05.838861 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_22bc40c64b7d4c8f8040ce17b7580930 [id=acb43ab939ff46428a795f618c6750dd]). Expected version 2 got 0
I20240503 14:12:05.838951 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25c852aa7aa4fbfafdacec43f5bc426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.840999 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 65853212c2404597b015ddc8ea99ee47 (DEFAULT_TABLE table=default.Table_17677d89d0664bb4b835cd1c24a9f27c [id=12558f7a877545f0900925e9c4948420]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.841521 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65853212c2404597b015ddc8ea99ee47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.841871 28184 tablet_service.cc:1467] Processing CreateTablet for tablet df63f75c157a4e928d4dbbda06cfa5ce (DEFAULT_TABLE table=default.Table_a9eb6e575b54421b8692dd6fc829eaf0 [id=d29fe68058a14e178e42a10043a91738]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.843436 28187 tablet_service.cc:1467] Processing CreateTablet for tablet c924a74ee836406fb3157baef6f4c901 (DEFAULT_TABLE table=default.Table_b76d17fed23441d3acf329eaa3d64853 [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.843927 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c924a74ee836406fb3157baef6f4c901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.844100 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_db4631ebdf2b4a0498017f281e91a22e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.844430 28332 tablet_service.cc:1467] Processing CreateTablet for tablet a3023785b11c4ad7b8757a715f0952c5 (DEFAULT_TABLE table=default.Table_2e0a149bc29c4f2fac86f41074f45afa [id=750bb7e271c241d2bbf14bd91dbc42d0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:05.844774 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db4631ebdf2b4a0498017f281e91a22e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.844980 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3023785b11c4ad7b8757a715f0952c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.847714 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 06fb79fcf30d485798c64c815e372dc1 (DEFAULT_TABLE table=default.Table_cef464f155254da3a6e7a33965f98e2d [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.847792 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 95d9985eecda4f3194c3251216aa58ee (DEFAULT_TABLE table=default.Table_627b41e76476426da74d6722301bc59f [id=97e7cb47edc44a5e9041be7ad4bc6f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.848254 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fb79fcf30d485798c64c815e372dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.848330 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2a035529f5284bd0b116e577ac353e9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:05.848462 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 9011de10a7ff497fb35ccb3ada7315ef (DEFAULT_TABLE table=default.Table_42bdb5d25c3e4d778e5c09ceb819bec4 [id=56e84e4023b242fc81a4e5c27c5ccd11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.848892 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9011de10a7ff497fb35ccb3ada7315ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:05.848927 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a035529f5284bd0b116e577ac353e9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.850811 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 1603e2360041427f8635782e812b6431 (DEFAULT_TABLE table=default.Table_d7bd421e71724c8a89b7cd478f447a3b [id=e0a3f887fca8477ba19c4ed3738b1955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.851313 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1603e2360041427f8635782e812b6431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.851625 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 (DEFAULT_TABLE table=default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.852074 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5775006e9f8f4612aa70dfb316de9ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.856510 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 0e680f189add4b5c96de4046321aad2d (DEFAULT_TABLE table=default.Table_7ad492405f92420ba1dd8dcf793d915c [id=d1088a94cd0442c9bd9837bf9f0e1ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.860020 28319 tablet_service.cc:1467] Processing CreateTablet for tablet df63f75c157a4e928d4dbbda06cfa5ce (DEFAULT_TABLE table=default.Table_a9eb6e575b54421b8692dd6fc829eaf0 [id=d29fe68058a14e178e42a10043a91738]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.860473 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 (DEFAULT_TABLE table=default.Table_77748b64009a403a8a014fa47bff52b4 [id=bc70f87f0df24f569c3cc1c65a86f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.863759 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 95d9985eecda4f3194c3251216aa58ee (DEFAULT_TABLE table=default.Table_627b41e76476426da74d6722301bc59f [id=97e7cb47edc44a5e9041be7ad4bc6f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.865672 28322 tablet_service.cc:1467] Processing CreateTablet for tablet c924a74ee836406fb3157baef6f4c901 (DEFAULT_TABLE table=default.Table_b76d17fed23441d3acf329eaa3d64853 [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.866827 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 7218cefb13ea4983b2111ed5fd0c997b (DEFAULT_TABLE table=default.Table_2e135732daa240d7b537d589e0256892 [id=3a24797cc53b439da44cc77025fa9f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.867292 28182 tablet_service.cc:1467] Processing CreateTablet for tablet b7d4e4a1f0b14406be957426a45bf45d (DEFAULT_TABLE table=default.Table_d3937b734026471d8da300845379e2dd [id=27afd28af9ee45a9ad0aad92898ce78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.867681 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 (DEFAULT_TABLE table=default.Table_77748b64009a403a8a014fa47bff52b4 [id=bc70f87f0df24f569c3cc1c65a86f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.868124 28327 tablet_service.cc:1467] Processing CreateTablet for tablet b7d4e4a1f0b14406be957426a45bf45d (DEFAULT_TABLE table=default.Table_d3937b734026471d8da300845379e2dd [id=27afd28af9ee45a9ad0aad92898ce78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.868458 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 0e680f189add4b5c96de4046321aad2d (DEFAULT_TABLE table=default.Table_7ad492405f92420ba1dd8dcf793d915c [id=d1088a94cd0442c9bd9837bf9f0e1ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.868836 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 1603e2360041427f8635782e812b6431 (DEFAULT_TABLE table=default.Table_d7bd421e71724c8a89b7cd478f447a3b [id=e0a3f887fca8477ba19c4ed3738b1955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.869068 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 65853212c2404597b015ddc8ea99ee47 (DEFAULT_TABLE table=default.Table_17677d89d0664bb4b835cd1c24a9f27c [id=12558f7a877545f0900925e9c4948420]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.869972 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e680f189add4b5c96de4046321aad2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:05.870308 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 6371e68ee85a47cdb0e8e21f7124d94a (table default.Table_355c3b14ae4f480dba47c2c90dc9dba2 [id=d7eac407621f4b03b21439fc2b0f7dc3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6371e68ee85a47cdb0e8e21f7124d94a consensus state has no leader
I20240503 14:12:05.868068 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 7218cefb13ea4983b2111ed5fd0c997b (DEFAULT_TABLE table=default.Table_2e135732daa240d7b537d589e0256892 [id=3a24797cc53b439da44cc77025fa9f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.871438 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 (DEFAULT_TABLE table=default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.871942 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 9011de10a7ff497fb35ccb3ada7315ef (DEFAULT_TABLE table=default.Table_42bdb5d25c3e4d778e5c09ceb819bec4 [id=56e84e4023b242fc81a4e5c27c5ccd11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.872129 28190 tablet_service.cc:1467] Processing CreateTablet for tablet a3023785b11c4ad7b8757a715f0952c5 (DEFAULT_TABLE table=default.Table_2e0a149bc29c4f2fac86f41074f45afa [id=750bb7e271c241d2bbf14bd91dbc42d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.879680 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df63f75c157a4e928d4dbbda06cfa5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.880445 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 06fb79fcf30d485798c64c815e372dc1 (DEFAULT_TABLE table=default.Table_cef464f155254da3a6e7a33965f98e2d [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.880975 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f50ad403bd354559b2522a76af1fa722
I20240503 14:12:05.881597 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d4e4a1f0b14406be957426a45bf45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.881981 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d9985eecda4f3194c3251216aa58ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.882297 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5775006e9f8f4612aa70dfb316de9ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.882642 29092 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 14:12:05.884416 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65853212c2404597b015ddc8ea99ee47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.885531 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fb79fcf30d485798c64c815e372dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.886830 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_068cea3b090241c7980d0edb57c05aec" } modify_external_catalogs: true
I20240503 14:12:05.887106 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3023785b11c4ad7b8757a715f0952c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.887125 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_068cea3b090241c7980d0edb57c05aec" } modify_external_catalogs: true
I20240503 14:12:05.887861 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7218cefb13ea4983b2111ed5fd0c997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.887815 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f99793320ab94248abb70c15ee57ca02" } modify_external_catalogs: true
I20240503 14:12:05.888013 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f99793320ab94248abb70c15ee57ca02" } modify_external_catalogs: true
I20240503 14:12:05.888242 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1728b6f0e64661aaa8184ce51366d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.888427 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e680f189add4b5c96de4046321aad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.889331 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df63f75c157a4e928d4dbbda06cfa5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.889885 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4fa4898a059c4416b632f8de37cd94d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.890697 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fa4898a059c4416b632f8de37cd94d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.891196 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9011de10a7ff497fb35ccb3ada7315ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.891512 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet f50ad403bd354559b2522a76af1fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.891310 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet f50ad403bd354559b2522a76af1fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:05.894079 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1603e2360041427f8635782e812b6431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.898077 28941 consensus_queue.cc:1035] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 14:12:05.902212 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1728b6f0e64661aaa8184ce51366d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.907004 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7218cefb13ea4983b2111ed5fd0c997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.908030 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d9985eecda4f3194c3251216aa58ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.909165 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6a7aee99214df4b02858ddae025b14" candidate_term: 2 candidate_status { last_received { term: 1 index: 272 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:12:05.914248 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d4e4a1f0b14406be957426a45bf45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.920363 28661 ts_tablet_manager.cc:1888] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:05.920886 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet f51834695b8b4a729a088da81eee2e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.923545 28455 tablet_service.cc:1514] Processing DeleteTablet for tablet 4775ff1029e6449c83219030e3809984 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.924229 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e0522da637fa4e2e880f9e9c076d3505" } modify_external_catalogs: true
I20240503 14:12:05.924355 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe5d280a1654d80a2176c1def459e2e with delete_type TABLET_DATA_DELETED (replaced by tablet dda39f0323ec47799d3d44c96f3ba605) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.924530 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e0522da637fa4e2e880f9e9c076d3505" } modify_external_catalogs: true
I20240503 14:12:05.926251 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.927528 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet bacf2d1091ee458c961c691093ea65a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.927767 28661 log.cc:1196] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/2502a13e8cbc4a3abfed225f94c44d2f
W20240503 14:12:05.929592 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 351d5bfb12c34c5f9de90e2df1b36fc1 consensus state has no leader
I20240503 14:12:05.929692 28473 tablet_service.cc:1467] Processing CreateTablet for tablet df63f75c157a4e928d4dbbda06cfa5ce (DEFAULT_TABLE table=default.Table_a9eb6e575b54421b8692dd6fc829eaf0 [id=d29fe68058a14e178e42a10043a91738]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.930306 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df63f75c157a4e928d4dbbda06cfa5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.930193 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_80c5e40f9e494b2dbbbb7449a5b36569"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.930861 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80c5e40f9e494b2dbbbb7449a5b36569 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.932548 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 95d9985eecda4f3194c3251216aa58ee (DEFAULT_TABLE table=default.Table_627b41e76476426da74d6722301bc59f [id=97e7cb47edc44a5e9041be7ad4bc6f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.933049 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d9985eecda4f3194c3251216aa58ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.933336 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cef464f155254da3a6e7a33965f98e2d" } new_table_name: "default.Table_3668987e06c74fa6bc566d6e46f92879" modify_external_catalogs: true
I20240503 14:12:05.934700 28025 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e] alter complete (version 1)
I20240503 14:12:05.936043 28456 tablet_service.cc:1467] Processing CreateTablet for tablet c924a74ee836406fb3157baef6f4c901 (DEFAULT_TABLE table=default.Table_b76d17fed23441d3acf329eaa3d64853 [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.936275 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 37dc7d96357743089372baf71654b1e3
I20240503 14:12:05.936570 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c924a74ee836406fb3157baef6f4c901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.936872 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 37dc7d96357743089372baf71654b1e3 deleted and replaced by tablet 5b90e9563b4c4cef8ee7964eaa96e285
I20240503 14:12:05.939903 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f0de27b9aac24572a6433bbba3533f63" } new_table_name: "default.Table_35f666e8e6c54a82893c615d5ffd476d" modify_external_catalogs: true
I20240503 14:12:05.940526 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 1603e2360041427f8635782e812b6431 (DEFAULT_TABLE table=default.Table_d7bd421e71724c8a89b7cd478f447a3b [id=e0a3f887fca8477ba19c4ed3738b1955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.941004 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1603e2360041427f8635782e812b6431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.944150 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc7d96357743089372baf71654b1e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b90e9563b4c4cef8ee7964eaa96e285) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.944332 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 0e680f189add4b5c96de4046321aad2d (DEFAULT_TABLE table=default.Table_7ad492405f92420ba1dd8dcf793d915c [id=d1088a94cd0442c9bd9837bf9f0e1ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.944844 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e680f189add4b5c96de4046321aad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.948647 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 (DEFAULT_TABLE table=default.Table_77748b64009a403a8a014fa47bff52b4 [id=bc70f87f0df24f569c3cc1c65a86f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.949211 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1728b6f0e64661aaa8184ce51366d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.952543 28469 tablet_service.cc:1467] Processing CreateTablet for tablet b7d4e4a1f0b14406be957426a45bf45d (DEFAULT_TABLE table=default.Table_d3937b734026471d8da300845379e2dd [id=27afd28af9ee45a9ad0aad92898ce78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.953084 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d4e4a1f0b14406be957426a45bf45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.958026 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b052ceeaac3644e18e8f3cb1947a3863" } modify_external_catalogs: true
I20240503 14:12:05.958308 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b052ceeaac3644e18e8f3cb1947a3863" } modify_external_catalogs: true
I20240503 14:12:05.961882 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3e9c030a424740bb2fb641328adfe1 with delete_type TABLET_DATA_DELETED (replaced by tablet bd76060daab84c2f9f6727b8a9bb3a49) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.962159 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.962477 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet eb79edba14b646ef9be239b05a291d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:05.962919 28455 tablet_service.cc:1467] Processing CreateTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 (DEFAULT_TABLE table=default.Table_3e4ec426791c4b11b43d3c36260eedd4 [id=c14f7d47ddd64652aa553945f775aec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.965641 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25c852aa7aa4fbfafdacec43f5bc426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.967038 28661 ts_tablet_manager.cc:1909] T 2502a13e8cbc4a3abfed225f94c44d2f P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:05.967470 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 139) took 2157 ms. Trace:
I20240503 14:12:05.967588 28661 rpcz_store.cc:276] 0503 14:12:03.810402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.823639 (+ 13237us) service_pool.cc:225] Handling call
0503 14:12:05.565017 (+1741378us) ts_tablet_manager.cc:1120] Deleting tablet 2502a13e8cbc4a3abfed225f94c44d2f
0503 14:12:05.565028 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.600399 (+ 35371us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.856859 (+256460us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.967428 (+110569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1714829}
I20240503 14:12:05.968016 28661 tablet_replica.cc:330] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:05.968314 28661 raft_consensus.cc:2229] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:05.968542 28661 raft_consensus.cc:2258] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:05.971367 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_dc507e002d97466bae378a75112560a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:05.972110 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc507e002d97466bae378a75112560a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:05.972950 28661 ts_tablet_manager.cc:1875] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:05.975533 28889 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" instance_seqno: 1714745518079174) as {username='slave'} at 127.27.50.193:49069; Asking this server to re-register.
I20240503 14:12:05.980656 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ffbd3291c56b4c13813c80d6775afe6f
I20240503 14:12:05.980854 28033 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 7ecf2b70082c4ade8d1778db30ae6e18
W20240503 14:12:05.981799 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06fb79fcf30d485798c64c815e372dc1 consensus state has no leader
W20240503 14:12:05.982477 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f1aa939aced4556ae0f0de0e62d1f97 consensus state has no leader
I20240503 14:12:05.984738 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbd3291c56b4c13813c80d6775afe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:05.987196 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 7218cefb13ea4983b2111ed5fd0c997b (DEFAULT_TABLE table=default.Table_2e135732daa240d7b537d589e0256892 [id=3a24797cc53b439da44cc77025fa9f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.995333 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 65853212c2404597b015ddc8ea99ee47 (DEFAULT_TABLE table=default.Table_17677d89d0664bb4b835cd1c24a9f27c [id=12558f7a877545f0900925e9c4948420]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.997350 29030 raft_consensus.cc:482] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.999711 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c924a74ee836406fb3157baef6f4c901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.001113 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 06fb79fcf30d485798c64c815e372dc1 (DEFAULT_TABLE table=default.Table_cef464f155254da3a6e7a33965f98e2d [id=92e89b0168d14b9a9fac6d7fca6be214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.001344 28460 tablet_service.cc:1467] Processing CreateTablet for tablet a3023785b11c4ad7b8757a715f0952c5 (DEFAULT_TABLE table=default.Table_2e0a149bc29c4f2fac86f41074f45afa [id=750bb7e271c241d2bbf14bd91dbc42d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.001662 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 (DEFAULT_TABLE table=default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.002013 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet f50ad403bd354559b2522a76af1fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.003134 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7218cefb13ea4983b2111ed5fd0c997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.003935 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 9011de10a7ff497fb35ccb3ada7315ef (DEFAULT_TABLE table=default.Table_42bdb5d25c3e4d778e5c09ceb819bec4 [id=56e84e4023b242fc81a4e5c27c5ccd11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.003754 29030 raft_consensus.cc:504] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.004484 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 65cdd090bf0f493f8b92341369378532 (DEFAULT_TABLE table=default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c [id=d16dae55faab45f4bb6337e042735ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.004587 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc7d96357743089372baf71654b1e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b90e9563b4c4cef8ee7964eaa96e285) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.004940 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cdd090bf0f493f8b92341369378532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.006060 29030 leader_election.cc:288] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:06.008567 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5775006e9f8f4612aa70dfb316de9ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.008903 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fb79fcf30d485798c64c815e372dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.009330 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 65cdd090bf0f493f8b92341369378532 (DEFAULT_TABLE table=default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c [id=d16dae55faab45f4bb6337e042735ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.009774 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cdd090bf0f493f8b92341369378532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.009461 28941 raft_consensus.cc:482] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.012239 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbd3291c56b4c13813c80d6775afe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.012791 28263 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:06.013406 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c800e997e944158f7c839dd0e00d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.014003 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 140de4d395d54b7696062ed96b8aa89c (DEFAULT_TABLE table=default.Table_dbb3e0a033c84560a6458571881b25e5 [id=238df3dccaa54f4f956476ccb58b0816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.014523 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140de4d395d54b7696062ed96b8aa89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.017585 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0b255d3e3dc64fb69c2ef6ed389db6a6
I20240503 14:12:06.017894 28941 raft_consensus.cc:504] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.018738 28941 leader_election.cc:288] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:06.020646 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4775ff1029e6449c83219030e3809984 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.024266 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc7d96357743089372baf71654b1e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b90e9563b4c4cef8ee7964eaa96e285) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.026898 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 140de4d395d54b7696062ed96b8aa89c (DEFAULT_TABLE table=default.Table_dbb3e0a033c84560a6458571881b25e5 [id=238df3dccaa54f4f956476ccb58b0816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.031951 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9011de10a7ff497fb35ccb3ada7315ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.032269 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:06.032832 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 2404807f8500437ababca53f6388b636 (DEFAULT_TABLE table=default.Table_7d8f2c48beb5472fb05f0cbd475f7182 [id=a396cc6875ba41d29e801308d3f274d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.033388 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2404807f8500437ababca53f6388b636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.035485 28471 tablet_service.cc:1467] Processing CreateTablet for tablet daf5d153ab5142c0902cc6e96629f412 (DEFAULT_TABLE table=default.Table_9d8f6693297844bc907fc355a5561203 [id=3c43a0fcf51143179a1ed0e1b8e333c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.036391 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 2502a13e8cbc4a3abfed225f94c44d2f (table default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:06.038908 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140de4d395d54b7696062ed96b8aa89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.041363 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 00b52ea336524beea2f255c8720f7472 (DEFAULT_TABLE table=default.Table_cca3611a96024dd49da0111bbdf07abd [id=eb3d80381dc844dcac4ed2bc171565d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.041714 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 2404807f8500437ababca53f6388b636 (DEFAULT_TABLE table=default.Table_7d8f2c48beb5472fb05f0cbd475f7182 [id=a396cc6875ba41d29e801308d3f274d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.041980 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 00b52ea336524beea2f255c8720f7472 (DEFAULT_TABLE table=default.Table_cca3611a96024dd49da0111bbdf07abd [id=eb3d80381dc844dcac4ed2bc171565d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.043030 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0795253db548fb84c2d7bb3dc48335 (DEFAULT_TABLE table=default.Table_c355201f2d654091bbf81fd66561a942 [id=38e97234a2c74ca4bd6dafbb6cba5284]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.044504 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b52ea336524beea2f255c8720f7472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.044657 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0795253db548fb84c2d7bb3dc48335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.045281 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b52ea336524beea2f255c8720f7472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.046029 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2404807f8500437ababca53f6388b636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.048563 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf5d153ab5142c0902cc6e96629f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.049470 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5e7ed1c5e8934a81bfeba19fe7452080" } new_table_name: "default.Table_48701243e80449bf86e7e0c400cebdc4" modify_external_catalogs: true
I20240503 14:12:06.051470 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_78145a3b2c8543b5ab5a675ab4b08c91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.052255 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78145a3b2c8543b5ab5a675ab4b08c91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.053085 28178 tablet_service.cc:1467] Processing CreateTablet for tablet daf5d153ab5142c0902cc6e96629f412 (DEFAULT_TABLE table=default.Table_9d8f6693297844bc907fc355a5561203 [id=3c43a0fcf51143179a1ed0e1b8e333c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.053296 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65853212c2404597b015ddc8ea99ee47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.054879 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.057523 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_22bc40c64b7d4c8f8040ce17b7580930" } new_table_name: "default.Table_eef00a50c2ce4752ab195301f31eddef" modify_external_catalogs: true
I20240503 14:12:06.058051 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.060577 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3023785b11c4ad7b8757a715f0952c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.060526 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_990e0eda7c604928a02942994c0e4803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.061234 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_990e0eda7c604928a02942994c0e4803 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.070173 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 2331aec871824a238f3f64de217d7092 (DEFAULT_TABLE table=default.Table_9d9d27627b064475a2f86b5ddcf4236f [id=2d1971a7d562426886d85395dfa1b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.070605 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf5d153ab5142c0902cc6e96629f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.072136 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2331aec871824a238f3f64de217d7092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.077587 28600 tablet_bootstrap.cc:492] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:06.087621 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0926e523ef454a2089f4d40a328c1c6d consensus state has no leader
I20240503 14:12:06.088467 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3668987e06c74fa6bc566d6e46f92879" } modify_external_catalogs: true
I20240503 14:12:06.088696 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1751f775fe6943d9b5fcc3b113393a73" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:06.088788 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3668987e06c74fa6bc566d6e46f92879" } modify_external_catalogs: true
I20240503 14:12:06.090201 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9622df66720f484b8f46a671ad60bf5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.091035 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9622df66720f484b8f46a671ad60bf5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.094040 28603 ts_tablet_manager.cc:1909] T a2bfdcb8e2b241ca9a6922fd8491e27b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:06.094626 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4ab48fb090ce4f93b2694880b967e8ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:06.094954 28216 raft_consensus.cc:2454] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:06.095294 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ab48fb090ce4f93b2694880b967e8ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.106855 28873 consensus_queue.cc:1035] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.115005 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.115712 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7c5dfac02d4dfe932996d2a37fe031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.115793 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.115962 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_00017e9cb2c6403f8a862b8cb661979c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.116892 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00017e9cb2c6403f8a862b8cb661979c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.119693 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0795253db548fb84c2d7bb3dc48335 (DEFAULT_TABLE table=default.Table_c355201f2d654091bbf81fd66561a942 [id=38e97234a2c74ca4bd6dafbb6cba5284]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.120307 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0795253db548fb84c2d7bb3dc48335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.123921 28427 leader_election.cc:302] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:06.126348 28873 consensus_queue.cc:1035] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.139361 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d56c053e47bb4621b5a7d74ca45069cc" } new_table_name: "default.Table_a7c8453b862c49c6923695b05befc40b" modify_external_catalogs: true
I20240503 14:12:06.139753 29030 raft_consensus.cc:2781] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:06.142522 29030 raft_consensus.cc:482] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:06.142726 29030 raft_consensus.cc:3037] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.157053 28889 ts_manager.cc:194] Registered new tserver with Master: a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:06.165377 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbd3291c56b4c13813c80d6775afe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.165529 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 2 got 0
I20240503 14:12:06.167760 28600 tablet_bootstrap.cc:654] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.169195 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 2404807f8500437ababca53f6388b636 (DEFAULT_TABLE table=default.Table_7d8f2c48beb5472fb05f0cbd475f7182 [id=a396cc6875ba41d29e801308d3f274d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.169299 28031 catalog_manager.cc:5484] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:06.169735 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 140de4d395d54b7696062ed96b8aa89c (DEFAULT_TABLE table=default.Table_dbb3e0a033c84560a6458571881b25e5 [id=238df3dccaa54f4f956476ccb58b0816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.170054 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 (DEFAULT_TABLE table=default.Table_80c5e40f9e494b2dbbbb7449a5b36569 [id=07297b58e7ca4fb1bf0e050a341d2c9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.170404 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 65cdd090bf0f493f8b92341369378532 (DEFAULT_TABLE table=default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c [id=d16dae55faab45f4bb6337e042735ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.170523 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abcb8658fc34c939aa74cf0bd981ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.173157 28873 consensus_queue.cc:1035] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.176051 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_49cdd819727f4ad381e0aae8fba8d201"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.176811 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49cdd819727f4ad381e0aae8fba8d201 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.187392 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5cf8c9529d1f4adb9fbee8eb11985651" } modify_external_catalogs: true
I20240503 14:12:06.187719 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5cf8c9529d1f4adb9fbee8eb11985651" } modify_external_catalogs: true
I20240503 14:12:06.188493 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a7f90c0c383146e393eb5890a2ce9168"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.189217 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7f90c0c383146e393eb5890a2ce9168 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.192231 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b (DEFAULT_TABLE table=default.Table_db4631ebdf2b4a0498017f281e91a22e [id=c7144a78f2ae47599016aa64b6d0cbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.193689 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60206e8e73e84e409ed6aba222d9ff4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.179396 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 02dd4c914a4846bab3a00408247ec64a (DEFAULT_TABLE table=default.Table_4fa4898a059c4416b632f8de37cd94d1 [id=b6e0d36bceef43e7810af395e207d8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.198539 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 77f0081cc2cb48aaa368741109032fd1 (DEFAULT_TABLE table=default.Table_bc583d61a09e4e87b183960e67bbc8b3 [id=ddb1f8d9ceae4da98dcb259cde8eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.201869 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f0081cc2cb48aaa368741109032fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.206588 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dd4c914a4846bab3a00408247ec64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.216439 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 (DEFAULT_TABLE table=default.Table_dc507e002d97466bae378a75112560a8 [id=3a0db192d94f40c3b69f089100ad18ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.217950 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5e3649c71f4d4da710c69f85a82c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.219211 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 00b52ea336524beea2f255c8720f7472 (DEFAULT_TABLE table=default.Table_cca3611a96024dd49da0111bbdf07abd [id=eb3d80381dc844dcac4ed2bc171565d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.223145 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2404807f8500437ababca53f6388b636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.223747 28320 tablet_service.cc:1467] Processing CreateTablet for tablet daf5d153ab5142c0902cc6e96629f412 (DEFAULT_TABLE table=default.Table_9d8f6693297844bc907fc355a5561203 [id=3c43a0fcf51143179a1ed0e1b8e333c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.229082 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b52ea336524beea2f255c8720f7472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.191877 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 147) took 2405 ms. Trace:
I20240503 14:12:06.237905 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf5d153ab5142c0902cc6e96629f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.242491 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a (DEFAULT_TABLE table=default.Table_2a035529f5284bd0b116e577ac353e9e [id=23dabedfe8df4e9fa6123f2dd9b82376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.245229 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.246456 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140de4d395d54b7696062ed96b8aa89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.250849 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45eb624434a404d85dcd233721ebf68" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:06.252728 28889 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" instance_seqno: 1714745518759808) as {username='slave'} at 127.27.50.194:37429; Asking this server to re-register.
I20240503 14:12:06.244801 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.257618 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45eb624434a404d85dcd233721ebf68" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:06.257686 29085 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.435s user 0.000s sys 0.000s
I20240503 14:12:06.266276 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_40f2cb0b89114bc291d2474f9babca16" } new_table_name: "default.Table_a888de5f6a40467db886c3141d2a87cc" modify_external_catalogs: true
I20240503 14:12:06.268385 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c" } modify_external_catalogs: true
I20240503 14:12:06.268694 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c" } modify_external_catalogs: true
I20240503 14:12:06.270671 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_777d8101e5ba4707a324b1ea2b061d2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.271442 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_777d8101e5ba4707a324b1ea2b061d2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:06.273015 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1751f775fe6943d9b5fcc3b113393a73 consensus state has no leader
I20240503 14:12:06.273521 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4dd04c06a7a04769b9b6eef8faac57a1" } new_table_name: "default.Table_67ce7228ef934797ba768f947d2cb0aa" modify_external_catalogs: true
I20240503 14:12:06.248945 28603 rpcz_store.cc:276] 0503 14:12:03.689450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.689516 (+ 66us) service_pool.cc:225] Handling call
0503 14:12:05.713890 (+2024374us) ts_tablet_manager.cc:1120] Deleting tablet a2bfdcb8e2b241ca9a6922fd8491e27b
0503 14:12:05.713899 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.760921 (+ 47022us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:05.789013 (+ 28092us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.094489 (+305476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2015570}
I20240503 14:12:06.275712 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.280144 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cdd090bf0f493f8b92341369378532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.281904 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]) successfully deleted
I20240503 14:12:06.295655 28889 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" instance_seqno: 1714745519449060) as {username='slave'} at 127.27.50.195:36047; Asking this server to re-register.
I20240503 14:12:06.296519 28653 ts_tablet_manager.cc:1888] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:06.296638 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0795253db548fb84c2d7bb3dc48335 (DEFAULT_TABLE table=default.Table_c355201f2d654091bbf81fd66561a942 [id=38e97234a2c74ca4bd6dafbb6cba5284]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.297070 28653 log.cc:1196] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/4775ff1029e6449c83219030e3809984
I20240503 14:12:06.298861 27971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6a7aee99214df4b02858ddae025b14" candidate_term: 2 candidate_status { last_received { term: 1 index: 272 } } ignore_live_leader: false dest_uuid: "a10ddca38c26462eb221d6026e88c6cc" is_pre_election: true
I20240503 14:12:06.300033 28653 ts_tablet_manager.cc:1909] T 4775ff1029e6449c83219030e3809984 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:06.304145 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 117) took 3397 ms. Trace:
I20240503 14:12:06.304525 28653 rpcz_store.cc:276] 0503 14:12:02.906635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.906924 (+ 289us) service_pool.cc:225] Handling call
0503 14:12:05.766739 (+2859815us) ts_tablet_manager.cc:1120] Deleting tablet 4775ff1029e6449c83219030e3809984
0503 14:12:05.766753 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:05.804618 (+ 37865us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.296280 (+491662us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.303971 (+ 7691us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2859381}
I20240503 14:12:06.305424 28653 tablet_replica.cc:330] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:06.305884 28653 raft_consensus.cc:2229] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:06.306381 28653 raft_consensus.cc:2258] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:06.306308 28848 leader_election.cc:302] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb6a7aee99214df4b02858ddae025b14; no voters: a10ddca38c26462eb221d6026e88c6cc, c0690a516c89446d85c88cdf64fa4238
I20240503 14:12:06.306988 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 02dd4c914a4846bab3a00408247ec64a (DEFAULT_TABLE table=default.Table_4fa4898a059c4416b632f8de37cd94d1 [id=b6e0d36bceef43e7810af395e207d8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.308183 28653 ts_tablet_manager.cc:1875] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:06.313529 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1751f775fe6943d9b5fcc3b113393a73" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:06.316186 28956 raft_consensus.cc:482] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.317109 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.318743 28403 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:06.316462 28941 consensus_queue.cc:1035] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20240503 14:12:06.319691 28290 leader_election.cc:302] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:06.319977 28603 tablet_replica.cc:330] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:06.283696 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 2331aec871824a238f3f64de217d7092 (DEFAULT_TABLE table=default.Table_9d9d27627b064475a2f86b5ddcf4236f [id=2d1971a7d562426886d85395dfa1b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.326329 29078 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:06.327615 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 (DEFAULT_TABLE table=default.Table_dc507e002d97466bae378a75112560a8 [id=3a0db192d94f40c3b69f089100ad18ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.330550 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 06fb79fcf30d485798c64c815e372dc1
I20240503 14:12:06.331261 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0795253db548fb84c2d7bb3dc48335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.331744 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0c1dc7a948b74ac8808b2e85171a0996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.332484 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c1dc7a948b74ac8808b2e85171a0996 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.332760 28956 raft_consensus.cc:504] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.333616 28956 leader_election.cc:288] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:06.333935 28354 raft_consensus.cc:2454] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:06.333998 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet eee989ff083041dc8dcd2cd1b220780d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.334687 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 2331aec871824a238f3f64de217d7092 (DEFAULT_TABLE table=default.Table_9d9d27627b064475a2f86b5ddcf4236f [id=2d1971a7d562426886d85395dfa1b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.335876 28941 raft_consensus.cc:2726] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:06.336014 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2c55428ec66a488990244819570ff1bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:06.336658 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3e4ec426791c4b11b43d3c36260eedd4" } modify_external_catalogs: true
W20240503 14:12:06.336763 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c55428ec66a488990244819570ff1bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.336879 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3e4ec426791c4b11b43d3c36260eedd4" } modify_external_catalogs: true
W20240503 14:12:06.338335 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba consensus state has no leader
I20240503 14:12:06.338671 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_986d778bcd814c9c87dbd18d1f485cc9" } modify_external_catalogs: true
I20240503 14:12:06.338755 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9eb6e575b54421b8692dd6fc829eaf0" } new_table_name: "default.Table_11a3df4a63e54fa98b91c2a818269000" modify_external_catalogs: true
I20240503 14:12:06.338932 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_986d778bcd814c9c87dbd18d1f485cc9" } modify_external_catalogs: true
I20240503 14:12:06.295229 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (DEFAULT_TABLE table=default.Table_d57cf3dd2d394751bf1d54730635a8dd [id=11af424ea24549bc83f6203448ae619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.340163 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 65cdd090bf0f493f8b92341369378532
I20240503 14:12:06.340961 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9953322d44f34ce5ad1123ba2784b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.341583 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9953322d44f34ce5ad1123ba2784b66f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.342130 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 4775ff1029e6449c83219030e3809984 (table default.Table_2ad02256d653464e9528725b5e37c72c [id=36785e97f5c94a15bf717d499a33f558]) successfully deleted
I20240503 14:12:06.347508 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b90e9563b4c4cef8ee7964eaa96e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.354411 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (DEFAULT_TABLE table=default.Table_d57cf3dd2d394751bf1d54730635a8dd [id=11af424ea24549bc83f6203448ae619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.354816 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8ba629915ef44e0882beee2612e285c1
I20240503 14:12:06.356729 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b25c852aa7aa4fbfafdacec43f5bc426
I20240503 14:12:06.360368 28195 tablet_service.cc:1467] Processing CreateTablet for tablet e6c90869b5034128864faaf05697c230 (DEFAULT_TABLE table=default.Table_c6f1f5b922f549679f50474fd6412cfc [id=47ade8624b00424d8fc04dbca5174ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.361708 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2331aec871824a238f3f64de217d7092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.362826 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:06.364748 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df63f75c157a4e928d4dbbda06cfa5ce consensus state has no leader
I20240503 14:12:06.364538 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 77f0081cc2cb48aaa368741109032fd1 (DEFAULT_TABLE table=default.Table_bc583d61a09e4e87b183960e67bbc8b3 [id=ddb1f8d9ceae4da98dcb259cde8eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.365294 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fb79fcf30d485798c64c815e372dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.365347 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 (DEFAULT_TABLE table=default.Table_80c5e40f9e494b2dbbbb7449a5b36569 [id=07297b58e7ca4fb1bf0e050a341d2c9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.365576 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 (DEFAULT_TABLE table=default.Table_c7035b3f783948498dfb7bd5ed14aad8 [id=15ae3079abdd4069aaeb84e1d85da988]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.366145 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 75dafd98b05d4c63bbf30165a2ac8d1e
I20240503 14:12:06.366161 29030 raft_consensus.cc:504] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.366935 29030 leader_election.cc:288] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:06.367169 28604 tablet_bootstrap.cc:492] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:06.367556 28604 ts_tablet_manager.cc:1367] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.725s user 0.018s sys 0.018s
I20240503 14:12:06.369601 28604 raft_consensus.cc:348] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.370117 28604 raft_consensus.cc:374] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.370450 28604 raft_consensus.cc:729] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:06.371315 29096 log.cc:924] Time spent T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240503 14:12:06.371508 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a (DEFAULT_TABLE table=default.Table_2a035529f5284bd0b116e577ac353e9e [id=23dabedfe8df4e9fa6123f2dd9b82376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.372467 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a7aefabd0c3841618e68382b8fa41a9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.373219 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7aefabd0c3841618e68382b8fa41a9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.371374 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5e3649c71f4d4da710c69f85a82c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.374172 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c9442a77363b44c584467e25ecc9878f" } new_table_name: "default.Table_dc4e1cabe9984508adddb6cbaf6bd934" modify_external_catalogs: true
I20240503 14:12:06.375835 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dd4c914a4846bab3a00408247ec64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.377087 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 (DEFAULT_TABLE table=default.Table_c7035b3f783948498dfb7bd5ed14aad8 [id=15ae3079abdd4069aaeb84e1d85da988]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.380224 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_eef00a50c2ce4752ab195301f31eddef [id=acb43ab939ff46428a795f618c6750dd]). Expected version 3 got 0
I20240503 14:12:06.370963 28604 consensus_queue.cc:260] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.388744 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.389611 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b (DEFAULT_TABLE table=default.Table_db4631ebdf2b4a0498017f281e91a22e [id=c7144a78f2ae47599016aa64b6d0cbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.390394 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c97a43bd714da68c1a77408f3d1cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.391660 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b90e9563b4c4cef8ee7964eaa96e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.392041 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abcb8658fc34c939aa74cf0bd981ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.392652 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f0081cc2cb48aaa368741109032fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.394013 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7c5dfac02d4dfe932996d2a37fe031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.394723 28603 raft_consensus.cc:2229] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:06.395192 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3703ab7db61b4c9ead3058bfdc123189 consensus state has no leader
I20240503 14:12:06.397709 28661 ts_tablet_manager.cc:1888] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:06.398923 28661 log.cc:1196] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/92c800e997e944158f7c839dd0e00d44
I20240503 14:12:06.399506 28661 ts_tablet_manager.cc:1909] T 92c800e997e944158f7c839dd0e00d44 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:06.403358 28468 tablet_service.cc:1467] Processing CreateTablet for tablet e6c90869b5034128864faaf05697c230 (DEFAULT_TABLE table=default.Table_c6f1f5b922f549679f50474fd6412cfc [id=47ade8624b00424d8fc04dbca5174ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.405774 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60206e8e73e84e409ed6aba222d9ff4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.406780 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.409965 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:06.412277 28325 consensus_meta.cc:287] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Time spent flushing consensus metadata: real 0.502s user 0.004s sys 0.000s
I20240503 14:12:06.413697 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 (DEFAULT_TABLE table=default.Table_80c5e40f9e494b2dbbbb7449a5b36569 [id=07297b58e7ca4fb1bf0e050a341d2c9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.415190 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fb79fcf30d485798c64c815e372dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.416814 28603 raft_consensus.cc:2258] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:06.420509 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 02dd4c914a4846bab3a00408247ec64a (DEFAULT_TABLE table=default.Table_4fa4898a059c4416b632f8de37cd94d1 [id=b6e0d36bceef43e7810af395e207d8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.420904 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c90869b5034128864faaf05697c230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.425073 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b (DEFAULT_TABLE table=default.Table_db4631ebdf2b4a0498017f281e91a22e [id=c7144a78f2ae47599016aa64b6d0cbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.429872 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 2 got 0
I20240503 14:12:06.441287 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 140) took 2614 ms. Trace:
I20240503 14:12:06.441659 28661 rpcz_store.cc:276] 0503 14:12:03.815906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.850234 (+ 34328us) service_pool.cc:225] Handling call
0503 14:12:05.967929 (+2117695us) ts_tablet_manager.cc:1120] Deleting tablet 92c800e997e944158f7c839dd0e00d44
0503 14:12:05.967938 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:06.083582 (+115644us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.397567 (+313985us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.429933 (+ 32366us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1637092}
I20240503 14:12:06.442353 28661 tablet_replica.cc:330] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:06.452780 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cdd090bf0f493f8b92341369378532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.456796 28604 ts_tablet_manager.cc:1398] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.058s user 0.004s sys 0.000s
I20240503 14:12:06.459730 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 (DEFAULT_TABLE table=default.Table_dc507e002d97466bae378a75112560a8 [id=3a0db192d94f40c3b69f089100ad18ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.469871 28661 raft_consensus.cc:2229] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:06.476745 28661 raft_consensus.cc:2258] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:06.477694 28661 ts_tablet_manager.cc:1875] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:06.479943 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4929b450373c4b689562f4a1d9e4b4f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.480947 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4929b450373c4b689562f4a1d9e4b4f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.481855 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fc86c8af96ac4a70906e1a764f52c59b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.482730 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc86c8af96ac4a70906e1a764f52c59b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.483250 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dc4e1cabe9984508adddb6cbaf6bd934" } new_table_name: "default.Table_fb279b38e7764d3bac7f2aa1667ca58e" modify_external_catalogs: true
I20240503 14:12:06.491153 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.491870 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.492528 28604 tablet_bootstrap.cc:492] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:06.499698 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3703ab7db61b4c9ead3058bfdc123189 consensus state has no leader
I20240503 14:12:06.501323 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a (DEFAULT_TABLE table=default.Table_2a035529f5284bd0b116e577ac353e9e [id=23dabedfe8df4e9fa6123f2dd9b82376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.507493 28603 ts_tablet_manager.cc:1875] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:06.513854 28336 tablet_service.cc:1467] Processing CreateTablet for tablet e6c90869b5034128864faaf05697c230 (DEFAULT_TABLE table=default.Table_c6f1f5b922f549679f50474fd6412cfc [id=47ade8624b00424d8fc04dbca5174ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.526052 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 77f0081cc2cb48aaa368741109032fd1 (DEFAULT_TABLE table=default.Table_bc583d61a09e4e87b183960e67bbc8b3 [id=ddb1f8d9ceae4da98dcb259cde8eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:06.531303 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 168033us
I20240503 14:12:06.531435 28614 tablet_bootstrap.cc:492] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:06.531253 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2331aec871824a238f3f64de217d7092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.534471 28889 ts_manager.cc:194] Registered new tserver with Master: d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:06.536921 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7c5dfac02d4dfe932996d2a37fe031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.536936 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (DEFAULT_TABLE table=default.Table_d57cf3dd2d394751bf1d54730635a8dd [id=11af424ea24549bc83f6203448ae619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.541208 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5e3649c71f4d4da710c69f85a82c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.543392 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 (DEFAULT_TABLE table=default.Table_c7035b3f783948498dfb7bd5ed14aad8 [id=15ae3079abdd4069aaeb84e1d85da988]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.546838 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_78145a3b2c8543b5ab5a675ab4b08c91" } modify_external_catalogs: true
I20240503 14:12:06.547195 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_78145a3b2c8543b5ab5a675ab4b08c91" } modify_external_catalogs: true
I20240503 14:12:06.548928 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0c1dc7a948b74ac8808b2e85171a0996" } new_table_name: "default.Table_89068da455a94e24972d9924fdab0795" modify_external_catalogs: true
I20240503 14:12:06.552332 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.553575 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abcb8658fc34c939aa74cf0bd981ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.554800 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60206e8e73e84e409ed6aba222d9ff4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.555895 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7b9e454770964e4594e5d043b06c2e78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:06.556250 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c97a43bd714da68c1a77408f3d1cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.556305 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_bc583d61a09e4e87b183960e67bbc8b3" } new_table_name: "default.Table_6ee87d5c5b1b4bde9f21a38ab19981d7" modify_external_catalogs: true
W20240503 14:12:06.556704 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b9e454770964e4594e5d043b06c2e78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.556968 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dd4c914a4846bab3a00408247ec64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.558079 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_00e404138f154498b730db1de48ded05" } modify_external_catalogs: true
I20240503 14:12:06.558272 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b90e9563b4c4cef8ee7964eaa96e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.558341 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_00e404138f154498b730db1de48ded05" } modify_external_catalogs: true
I20240503 14:12:06.559149 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cc68335896e1489789a7ca2101fa68ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.559829 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc68335896e1489789a7ca2101fa68ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.562062 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c90869b5034128864faaf05697c230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.571516 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.574230 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 5d69cc969ea949a684a7e5aae237c13e (DEFAULT_TABLE table=default.Table_990e0eda7c604928a02942994c0e4803 [id=0114799f37994a3396ee46fd1f2e98b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.574432 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dc507e002d97466bae378a75112560a8" } modify_external_catalogs: true
I20240503 14:12:06.574717 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dc507e002d97466bae378a75112560a8" } modify_external_catalogs: true
I20240503 14:12:06.575433 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_80c5e40f9e494b2dbbbb7449a5b36569" } modify_external_catalogs: true
I20240503 14:12:06.575678 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_80c5e40f9e494b2dbbbb7449a5b36569" } modify_external_catalogs: true
I20240503 14:12:06.576260 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet e355aafb2f5b4337bedaac6fe5539dc2 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:06.576488 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 2f89fca0480d4bf28160384ae0421bce (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.580269 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_34ff5e18a1334c7faa054816f0837ee2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.581022 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34ff5e18a1334c7faa054816f0837ee2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:06.583036 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed4dc732bcb44222b4b15e5d77defe33 consensus state has no leader
I20240503 14:12:06.583711 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4d3ef3849e8a4cc3a225bdfea8c88884"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.584478 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d3ef3849e8a4cc3a225bdfea8c88884 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.586393 28467 tablet_service.cc:1467] Processing CreateTablet for tablet c5297664a4d943c28adfb6ac4f061680 (DEFAULT_TABLE table=default.Table_78145a3b2c8543b5ab5a675ab4b08c91 [id=99ed67dc285f4dc69a9a9c6cec10b729]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.588359 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 495190fca06a40a386c8215eb2195920 (DEFAULT_TABLE table=default.Table_49cdd819727f4ad381e0aae8fba8d201 [id=f7e1e9a608664cd6967ef0521d9cb671]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:06.589344 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77f0081cc2cb48aaa368741109032fd1 consensus state has no leader
I20240503 14:12:06.589929 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_466e18e3e6f346e798971d5b48a74866" } new_table_name: "default.Table_ce1f05b05bd3429eacedb14a5f34b7da" modify_external_catalogs: true
I20240503 14:12:06.591503 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d69cc969ea949a684a7e5aae237c13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.591459 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fe566740b43c42658f0a0ff2de59038a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:06.532008 28614 ts_tablet_manager.cc:1367] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.746s user 0.008s sys 0.008s
I20240503 14:12:06.594338 28456 tablet_service.cc:1467] Processing CreateTablet for tablet bf21f6234dd64cf8a32f1599fe50746e (DEFAULT_TABLE table=default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.596843 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.597607 28466 tablet_service.cc:1467] Processing CreateTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 (DEFAULT_TABLE table=default.Table_9953322d44f34ce5ad1123ba2784b66f [id=5ae48705fb644bddba8e69d66c640a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.598079 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 903f11a608b1483887d9b5234a404470 (DEFAULT_TABLE table=default.Table_00017e9cb2c6403f8a862b8cb661979c [id=32be054076954aed9265888ca62e510b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.598583 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet eee989ff083041dc8dcd2cd1b220780d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.602088 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c90869b5034128864faaf05697c230. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:06.603299 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe566740b43c42658f0a0ff2de59038a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.607913 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f89fca0480d4bf28160384ae0421bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.615912 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9a954e776a184d819708a62a99ed94fc" } modify_external_catalogs: true
I20240503 14:12:06.616237 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9a954e776a184d819708a62a99ed94fc" } modify_external_catalogs: true
I20240503 14:12:06.618871 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_68a22cfee7f644fcba9b9c3e34e6205c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.619550 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68a22cfee7f644fcba9b9c3e34e6205c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.619845 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 (DEFAULT_TABLE table=default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.621495 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c97a43bd714da68c1a77408f3d1cd1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:06.626890 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a32b6995818041778617a0d6bf936c8c consensus state has no leader
I20240503 14:12:06.627678 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2a035529f5284bd0b116e577ac353e9e" } new_table_name: "default.Table_79df8c9603ae456799a73fb5badd18e9" modify_external_catalogs: true
I20240503 14:12:06.633044 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 92c800e997e944158f7c839dd0e00d44 (table default.Table_5f10f98658754315a6cc9b1d7ffc19cb [id=d9b865dc744148df9fc1ab6e79c5043d]) successfully deleted
I20240503 14:12:06.636505 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_222131e1d2a848758bd7e6b5a7498f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.652863 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_222131e1d2a848758bd7e6b5a7498f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.660693 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba629915ef44e0882beee2612e285c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:06.677528 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a consensus state has no leader
I20240503 14:12:06.678766 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fb79fcf30d485798c64c815e372dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.679505 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.685063 28604 tablet_bootstrap.cc:654] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.685133 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba629915ef44e0882beee2612e285c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.685142 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1111e7c3fe431ea653321d1dee388c (DEFAULT_TABLE table=default.Table_4ab48fb090ce4f93b2694880b967e8ef [id=ed3e08c76b124e2b90bb3cd69e81f4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.685452 28540 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:06.686084 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cdd090bf0f493f8b92341369378532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:06.688117 28290 leader_election.cc:341] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 4775ff1029e6449c83219030e3809984
I20240503 14:12:06.694243 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cdd090bf0f493f8b92341369378532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.695191 28941 raft_consensus.cc:482] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.695475 28956 raft_consensus.cc:482] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.695631 28956 raft_consensus.cc:504] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.696302 28327 tablet_service.cc:1467] Processing CreateTablet for tablet c5297664a4d943c28adfb6ac4f061680 (DEFAULT_TABLE table=default.Table_78145a3b2c8543b5ab5a675ab4b08c91 [id=99ed67dc285f4dc69a9a9c6cec10b729]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.700580 28941 raft_consensus.cc:504] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.701460 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:06.702291 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_037096975b83431e99d595ed4e62d706" } new_table_name: "default.Table_701e9c2bdea449d49f86dffb86090877" modify_external_catalogs: true
I20240503 14:12:06.703420 28468 tablet_service.cc:1467] Processing CreateTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 (DEFAULT_TABLE table=default.Table_0c1dc7a948b74ac8808b2e85171a0996 [id=a6a9ad0f171448c2a3d3401e02ec62c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.704547 28941 leader_election.cc:288] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:06.707216 28600 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.010s sys 0.029s
I20240503 14:12:06.708478 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_32b3856710f446139fa555cd42254eef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:06.709280 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32b3856710f446139fa555cd42254eef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:06.709847 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495190fca06a40a386c8215eb2195920. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:06.709666 28600 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.539s user 0.010s sys 0.030s
I20240503 14:12:06.692988 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6c998e518541248a87e4c1551ec080 (DEFAULT_TABLE table=default.Table_a7aefabd0c3841618e68382b8fa41a9a [id=c23a0b9e98f949559497aea2a4dab24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.716495 28319 tablet_service.cc:1467] Processing CreateTablet for tablet 5d69cc969ea949a684a7e5aae237c13e (DEFAULT_TABLE table=default.Table_990e0eda7c604928a02942994c0e4803 [id=0114799f37994a3396ee46fd1f2e98b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.717098 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f0081cc2cb48aaa368741109032fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.719470 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf21f6234dd64cf8a32f1599fe50746e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.732373 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:06.744534 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ba629915ef44e0882beee2612e285c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.745942 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.746783 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:06.748405 28600 tablet_bootstrap.cc:492] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:06.748863 28600 ts_tablet_manager.cc:1367] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.694s user 0.012s sys 0.033s
I20240503 14:12:06.750589 28600 raft_consensus.cc:348] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:06.751677 28150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.019s sys 0.029s
I20240503 14:12:06.752664 28187 tablet_service.cc:1467] Processing CreateTablet for tablet c5297664a4d943c28adfb6ac4f061680 (DEFAULT_TABLE table=default.Table_78145a3b2c8543b5ab5a675ab4b08c91 [id=99ed67dc285f4dc69a9a9c6cec10b729]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.755120 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1111e7c3fe431ea653321d1dee388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.755493 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a92fc3833e49aaab8c5415de9bb685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.758162 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 5a081ba59ad44890bf2f94874c034277 (DEFAULT_TABLE table=default.Table_a7f90c0c383146e393eb5890a2ce9168 [id=114dfd39b4ea4958a08161a34e4a1f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.762010 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903f11a608b1483887d9b5234a404470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.763876 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 903f11a608b1483887d9b5234a404470 (DEFAULT_TABLE table=default.Table_00017e9cb2c6403f8a862b8cb661979c [id=32be054076954aed9265888ca62e510b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.764631 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5297664a4d943c28adfb6ac4f061680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.766072 28889 ts_manager.cc:194] Registered new tserver with Master: d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:06.771965 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 2f89fca0480d4bf28160384ae0421bce (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.772671 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1751f775fe6943d9b5fcc3b113393a73" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:06.773517 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5297664a4d943c28adfb6ac4f061680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.774061 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 8410e8b6b511403ea609b1353855937b (DEFAULT_TABLE table=default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280 [id=05881d1136974066b34909e625c970a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.716334 28330 tablet_service.cc:1467] Processing CreateTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 (DEFAULT_TABLE table=default.Table_9953322d44f34ce5ad1123ba2784b66f [id=5ae48705fb644bddba8e69d66c640a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.774813 28354 raft_consensus.cc:3037] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.775261 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 2221218847554cbd9d3242a9a6431e46 (DEFAULT_TABLE table=default.Table_4929b450373c4b689562f4a1d9e4b4f2 [id=7b85025507014faa9ce25ba44310b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.775655 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 2f89fca0480d4bf28160384ae0421bce (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.776611 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5297664a4d943c28adfb6ac4f061680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.777038 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 5d69cc969ea949a684a7e5aae237c13e (DEFAULT_TABLE table=default.Table_990e0eda7c604928a02942994c0e4803 [id=0114799f37994a3396ee46fd1f2e98b5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:06.777575 28150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.019s sys 0.029s
I20240503 14:12:06.779172 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b6c998e518541248a87e4c1551ec080. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:06.779146 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 606314us
I20240503 14:12:06.785312 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 495190fca06a40a386c8215eb2195920 (DEFAULT_TABLE table=default.Table_49cdd819727f4ad381e0aae8fba8d201 [id=f7e1e9a608664cd6967ef0521d9cb671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.785523 29101 raft_consensus.cc:482] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.785733 29101 raft_consensus.cc:504] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.785969 28182 tablet_service.cc:1467] Processing CreateTablet for tablet bf21f6234dd64cf8a32f1599fe50746e (DEFAULT_TABLE table=default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.786510 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903f11a608b1483887d9b5234a404470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.786425 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 (DEFAULT_TABLE table=default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.786695 29101 leader_election.cc:288] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:06.786875 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af0342e0e0b47499a9322b5b2cb3d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.789738 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8410e8b6b511403ea609b1353855937b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.792044 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495190fca06a40a386c8215eb2195920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.797627 28661 ts_tablet_manager.cc:1888] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:06.799147 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a081ba59ad44890bf2f94874c034277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.800410 28661 log.cc:1196] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/559342d357d64f9eb9e3ab42ccb75942
I20240503 14:12:06.801980 28188 tablet_service.cc:1467] Processing CreateTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 (DEFAULT_TABLE table=default.Table_9953322d44f34ce5ad1123ba2784b66f [id=5ae48705fb644bddba8e69d66c640a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.803260 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 495190fca06a40a386c8215eb2195920 (DEFAULT_TABLE table=default.Table_49cdd819727f4ad381e0aae8fba8d201 [id=f7e1e9a608664cd6967ef0521d9cb671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.804838 28956 leader_election.cc:288] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:06.805116 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a92fc3833e49aaab8c5415de9bb685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.805615 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d69cc969ea949a684a7e5aae237c13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.806685 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f89fca0480d4bf28160384ae0421bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.808753 28661 ts_tablet_manager.cc:1909] T 559342d357d64f9eb9e3ab42ccb75942 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:06.809803 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 142) took 2967 ms. Trace:
I20240503 14:12:06.810163 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf21f6234dd64cf8a32f1599fe50746e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.815567 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f89fca0480d4bf28160384ae0421bce. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:06.824527 28290 leader_election.cc:341] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 4775ff1029e6449c83219030e3809984
I20240503 14:12:06.824889 28290 leader_election.cc:302] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:06.827628 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbabe5beb9a42c69cb64e1b364d75a1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:06.828188 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee989ff083041dc8dcd2cd1b220780d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:06.829236 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee989ff083041dc8dcd2cd1b220780d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:06.829479 28216 raft_consensus.cc:2454] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:06.829695 28941 raft_consensus.cc:2726] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:06.830130 28290 leader_election.cc:341] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 9dbabe5beb9a42c69cb64e1b364d75a1
I20240503 14:12:06.830343 28290 leader_election.cc:302] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:06.830899 28839 raft_consensus.cc:2726] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:06.836431 28357 raft_consensus.cc:1261] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.839071 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495190fca06a40a386c8215eb2195920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.840708 28614 raft_consensus.cc:348] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:06.845268 28614 raft_consensus.cc:374] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.846341 28614 raft_consensus.cc:729] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.860437 28600 raft_consensus.cc:374] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.860657 28600 raft_consensus.cc:729] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.861151 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2221218847554cbd9d3242a9a6431e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.861104 28600 consensus_queue.cc:260] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:06.861945 28600 ts_tablet_manager.cc:1398] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.113s user 0.002s sys 0.000s
I20240503 14:12:06.862452 28600 tablet_bootstrap.cc:492] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:06.862537 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af0342e0e0b47499a9322b5b2cb3d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.863953 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4dc732bcb44222b4b15e5d77defe33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.861542 28661 rpcz_store.cc:276] 0503 14:12:03.841950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.865309 (+ 23359us) service_pool.cc:225] Handling call
0503 14:12:06.442187 (+2576878us) ts_tablet_manager.cc:1120] Deleting tablet 559342d357d64f9eb9e3ab42ccb75942
0503 14:12:06.442202 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:06.601253 (+159051us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.784170 (+182917us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.809754 (+ 25584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2110780}
I20240503 14:12:06.868198 28661 tablet_replica.cc:330] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:06.875718 28661 raft_consensus.cc:2229] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:06.876303 28661 raft_consensus.cc:2258] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:06.877336 28661 ts_tablet_manager.cc:1875] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:06.890164 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 559342d357d64f9eb9e3ab42ccb75942 (table default.Table_7d8e4d1ecd794871b9d73289f8f43c9a [id=c6805f8e2643425dbed90e5ebe7d3a4a]) successfully deleted
I20240503 14:12:06.892421 29030 raft_consensus.cc:482] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:06.892679 29030 raft_consensus.cc:504] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.903565 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 903f11a608b1483887d9b5234a404470 (DEFAULT_TABLE table=default.Table_00017e9cb2c6403f8a862b8cb661979c [id=32be054076954aed9265888ca62e510b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.909739 29092 catalog_manager.cc:1259] Loaded cluster ID: 09434ad4cf0745ecb5d037e3cd83df07
I20240503 14:12:06.910032 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 64bae128469f4e8bb2d743320731446a (DEFAULT_TABLE table=default.Table_fc86c8af96ac4a70906e1a764f52c59b [id=cb2015955266483fa60f3fc692731e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.913342 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bae128469f4e8bb2d743320731446a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.913319 29092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: loading cluster ID for follower catalog manager: success
I20240503 14:12:06.915897 28318 tablet_service.cc:1467] Processing CreateTablet for tablet bf21f6234dd64cf8a32f1599fe50746e (DEFAULT_TABLE table=default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.916303 29030 leader_election.cc:288] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:06.919800 28604 tablet_bootstrap.cc:492] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:06.922509 28604 ts_tablet_manager.cc:1367] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.463s user 0.016s sys 0.025s
W20240503 14:12:06.925599 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1751f775fe6943d9b5fcc3b113393a73 (table default.Table_a7c8453b862c49c6923695b05befc40b [id=9931742257d74df99c043e569a267ced], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1751f775fe6943d9b5fcc3b113393a73 consensus state has no leader
I20240503 14:12:06.928052 29092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: acquiring CA information for follower catalog manager: success
I20240503 14:12:06.919095 28354 raft_consensus.cc:2454] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:06.831118 28290 leader_election.cc:341] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:06.950870 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 106028us
I20240503 14:12:06.948920 28604 raft_consensus.cc:348] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.955919 28661 ts_tablet_manager.cc:1888] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:06.865296 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d69cc969ea949a684a7e5aae237c13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.875422 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6c998e518541248a87e4c1551ec080 (DEFAULT_TABLE table=default.Table_a7aefabd0c3841618e68382b8fa41a9a [id=c23a0b9e98f949559497aea2a4dab24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.876564 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 5a081ba59ad44890bf2f94874c034277 (DEFAULT_TABLE table=default.Table_a7f90c0c383146e393eb5890a2ce9168 [id=114dfd39b4ea4958a08161a34e4a1f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.877048 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 8410e8b6b511403ea609b1353855937b (DEFAULT_TABLE table=default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280 [id=05881d1136974066b34909e625c970a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.894668 28653 ts_tablet_manager.cc:1888] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:06.957087 28873 consensus_queue.cc:1035] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.969619 28604 raft_consensus.cc:374] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.969770 28604 raft_consensus.cc:729] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.970055 28604 consensus_queue.cc:260] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:06.970400 28661 log.cc:1196] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ab05178dc081480ba409aab7f143f849
I20240503 14:12:06.970772 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde0e59d911d4c1eb5125b1f869b396c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:06.970988 28661 ts_tablet_manager.cc:1909] T ab05178dc081480ba409aab7f143f849 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:06.972119 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903f11a608b1483887d9b5234a404470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.971958 28614 consensus_queue.cc:260] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:06.973361 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 141) took 3131 ms. Trace:
I20240503 14:12:06.973520 28661 rpcz_store.cc:276] 0503 14:12:03.841771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.864850 (+ 23079us) service_pool.cc:225] Handling call
0503 14:12:06.866451 (+3001601us) ts_tablet_manager.cc:1120] Deleting tablet ab05178dc081480ba409aab7f143f849
0503 14:12:06.866468 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:06.951120 (+ 84652us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.955766 (+ 4646us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.973251 (+ 17485us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1324,"tablet-delete.queue_time_us":2534618}
I20240503 14:12:06.974040 28661 tablet_replica.cc:330] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:06.974277 28661 raft_consensus.cc:2229] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:06.974573 28661 raft_consensus.cc:2258] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:06.975270 28661 ts_tablet_manager.cc:1875] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:06.976373 28604 ts_tablet_manager.cc:1398] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.053s user 0.000s sys 0.003s
I20240503 14:12:06.979146 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 5a081ba59ad44890bf2f94874c034277 (DEFAULT_TABLE table=default.Table_a7f90c0c383146e393eb5890a2ce9168 [id=114dfd39b4ea4958a08161a34e4a1f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:06.979588 28604 tablet_bootstrap.cc:492] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:06.981562 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf21f6234dd64cf8a32f1599fe50746e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.983872 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a92fc3833e49aaab8c5415de9bb685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.985605 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 2 got 0
I20240503 14:12:06.987236 28604 tablet_bootstrap.cc:654] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.987584 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 2 got 0
I20240503 14:12:06.988981 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a081ba59ad44890bf2f94874c034277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.991515 28032 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_eef00a50c2ce4752ab195301f31eddef [id=acb43ab939ff46428a795f618c6750dd]). Expected version 3 got 0
I20240503 14:12:06.994668 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 2 got 0
I20240503 14:12:06.995254 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_eef00a50c2ce4752ab195301f31eddef [id=acb43ab939ff46428a795f618c6750dd]). Expected version 3 got 0
I20240503 14:12:06.995690 28653 log.cc:1196] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ea3e9c030a424740bb2fb641328adfe1
I20240503 14:12:06.996238 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1111e7c3fe431ea653321d1dee388c (DEFAULT_TABLE table=default.Table_4ab48fb090ce4f93b2694880b967e8ef [id=ed3e08c76b124e2b90bb3cd69e81f4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.005913 28754 consensus_queue.cc:1035] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.006832 28754 consensus_queue.cc:1035] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:07.007301 28427 leader_election.cc:341] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a0e8b0eed57c4ad1bf6040507f6f8d4c
I20240503 14:12:07.022127 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b6c998e518541248a87e4c1551ec080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.020730 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6665f4b8f8e449a8bbd3d22231c667d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:07.025290 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1751f775fe6943d9b5fcc3b113393a73" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:07.025606 28212 raft_consensus.cc:3037] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.028445 28653 ts_tablet_manager.cc:1909] T ea3e9c030a424740bb2fb641328adfe1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:07.027853 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ab05178dc081480ba409aab7f143f849 (table default.Table_adb8e4f58ba042f0a57523a25228ca92 [id=37c31c087c65469f96f4322d84ba161f]) successfully deleted
W20240503 14:12:07.028635 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6665f4b8f8e449a8bbd3d22231c667d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.029896 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 118) took 4056 ms. Trace:
I20240503 14:12:07.030066 28661 ts_tablet_manager.cc:1888] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:07.030100 28653 rpcz_store.cc:276] 0503 14:12:02.973350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.973436 (+ 86us) service_pool.cc:225] Handling call
0503 14:12:06.305220 (+3331784us) ts_tablet_manager.cc:1120] Deleting tablet ea3e9c030a424740bb2fb641328adfe1
0503 14:12:06.305233 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:06.364779 (+ 59546us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:06.894439 (+529660us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.029846 (+135407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3331387}
I20240503 14:12:07.030658 28653 tablet_replica.cc:330] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:07.030859 28661 log.cc:1196] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a2bfdcb8e2b241ca9a6922fd8491e27b
I20240503 14:12:07.031082 28653 raft_consensus.cc:2229] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.031405 28653 raft_consensus.cc:2258] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.031862 28661 ts_tablet_manager.cc:1909] T a2bfdcb8e2b241ca9a6922fd8491e27b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:07.032269 28653 ts_tablet_manager.cc:1875] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.032545 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 2221218847554cbd9d3242a9a6431e46 (DEFAULT_TABLE table=default.Table_4929b450373c4b689562f4a1d9e4b4f2 [id=7b85025507014faa9ce25ba44310b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.033150 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 143) took 3191 ms. Trace:
I20240503 14:12:07.038002 28661 rpcz_store.cc:276] 0503 14:12:03.842062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.893796 (+ 51734us) service_pool.cc:225] Handling call
0503 14:12:06.973881 (+3080085us) ts_tablet_manager.cc:1120] Deleting tablet a2bfdcb8e2b241ca9a6922fd8491e27b
0503 14:12:06.973893 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.002811 (+ 28918us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.025186 (+ 22375us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.033104 (+ 7918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2641987}
I20240503 14:12:07.037797 28604 tablet_bootstrap.cc:492] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:07.039579 28604 ts_tablet_manager.cc:1367] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.009s
I20240503 14:12:07.040930 28604 raft_consensus.cc:348] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.041301 28604 raft_consensus.cc:374] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.041473 28604 raft_consensus.cc:729] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.041867 28604 consensus_queue.cc:260] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:07.042527 28150 consensus_peers.cc:485] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9dbabe5beb9a42c69cb64e1b364d75a1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:07.043839 29092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:12:07.045022 28604 ts_tablet_manager.cc:1398] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:12:07.045496 28604 tablet_bootstrap.cc:492] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:07.046216 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 146) took 3123 ms. Trace:
I20240503 14:12:07.049877 28661 rpcz_store.cc:276] 0503 14:12:03.922540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.927160 (+ 4620us) service_pool.cc:225] Handling call
0503 14:12:07.045874 (+3118714us) ts_tablet_manager.cc:1120] Deleting tablet 93462d2b1a474bd688d6266cd3b0695c
0503 14:12:07.045892 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.046142 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2713517}
I20240503 14:12:07.058960 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6c998e518541248a87e4c1551ec080 (DEFAULT_TABLE table=default.Table_a7aefabd0c3841618e68382b8fa41a9a [id=c23a0b9e98f949559497aea2a4dab24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.059381 28661 tablet_replica.cc:330] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:07.061136 28604 tablet_bootstrap.cc:654] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.065903 28614 ts_tablet_manager.cc:1398] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.380s user 0.005s sys 0.000s
I20240503 14:12:07.061381 28661 raft_consensus.cc:2229] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.066362 28661 raft_consensus.cc:2258] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.066404 28614 tablet_bootstrap.cc:492] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:07.067169 28661 ts_tablet_manager.cc:1875] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.070624 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 (DEFAULT_TABLE table=default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:07.071977 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 76114.6us
W20240503 14:12:07.074220 28290 consensus_peers.cc:485] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 559342d357d64f9eb9e3ab42ccb75942. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:07.078105 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8410e8b6b511403ea609b1353855937b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.078595 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2221218847554cbd9d3242a9a6431e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.081141 28182 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1111e7c3fe431ea653321d1dee388c (DEFAULT_TABLE table=default.Table_4ab48fb090ce4f93b2694880b967e8ef [id=ed3e08c76b124e2b90bb3cd69e81f4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.082412 28873 consensus_queue.cc:1035] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:07.082656 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 122672us
I20240503 14:12:07.083552 29107 raft_consensus.cc:482] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.083985 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1111e7c3fe431ea653321d1dee388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.084838 28754 consensus_queue.cc:1035] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.085117 29030 consensus_queue.cc:1035] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.086758 28427 leader_election.cc:302] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:07.087205 29030 raft_consensus.cc:2781] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.087388 29030 raft_consensus.cc:686] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:07.087764 29030 consensus_queue.cc:237] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.095044 28192 tablet_service.cc:1467] Processing CreateTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 (DEFAULT_TABLE table=default.Table_0c1dc7a948b74ac8808b2e85171a0996 [id=a6a9ad0f171448c2a3d3401e02ec62c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.100350 29107 raft_consensus.cc:504] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.106235 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde0e59d911d4c1eb5125b1f869b396c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:07.107324 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1c5e3649c71f4d4da710c69f85a82c25
I20240503 14:12:07.108439 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 50edc38cdbe74397aa4f4928d56d9fbc
I20240503 14:12:07.109596 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7abcb8658fc34c939aa74cf0bd981ae4
I20240503 14:12:07.109791 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 64bae128469f4e8bb2d743320731446a (DEFAULT_TABLE table=default.Table_fc86c8af96ac4a70906e1a764f52c59b [id=cb2015955266483fa60f3fc692731e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.110090 28356 raft_consensus.cc:3037] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.110628 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet c5297664a4d943c28adfb6ac4f061680
I20240503 14:12:07.111243 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 74df7781cc6a4dd6b6fd78a728ffd71b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:07.111725 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4dc732bcb44222b4b15e5d77defe33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.112337 28319 tablet_service.cc:1467] Processing CreateTablet for tablet 8410e8b6b511403ea609b1353855937b (DEFAULT_TABLE table=default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280 [id=05881d1136974066b34909e625c970a7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:07.112723 28321 consensus_meta.cc:287] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Time spent flushing consensus metadata: real 0.519s user 0.003s sys 0.007s
I20240503 14:12:07.116521 28212 raft_consensus.cc:2454] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:07.121672 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]) successfully deleted
I20240503 14:12:07.123884 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a081ba59ad44890bf2f94874c034277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.124212 28355 raft_consensus.cc:3037] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.125108 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8b0eed57c4ad1bf6040507f6f8d4c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:07.125367 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bae128469f4e8bb2d743320731446a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:07.126838 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 93462d2b1a474bd688d6266cd3b0695c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93462d2b1a474bd688d6266cd3b0695c already in progress: creating tablet
I20240503 14:12:07.128057 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af0342e0e0b47499a9322b5b2cb3d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.129513 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1111e7c3fe431ea653321d1dee388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.134358 28321 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58028 (request call id 200) took 1287 ms. Trace:
I20240503 14:12:07.134574 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e135b60660f448a7974520f2f87cf93a
I20240503 14:12:07.136185 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e355aafb2f5b4337bedaac6fe5539dc2
I20240503 14:12:07.136739 28025 catalog_manager.cc:6554] ReplaceTablet: tablet e355aafb2f5b4337bedaac6fe5539dc2 deleted and replaced by tablet 80cd09e769be4bee95bbef9c3c550451
I20240503 14:12:07.137037 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.137391 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.139345 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.131287 29107 leader_election.cc:288] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:07.140199 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet c5297664a4d943c28adfb6ac4f061680 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.139643 28661 ts_tablet_manager.cc:1888] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:07.143631 28661 log.cc:1196] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7de7832122804eada175f710cc5bab5e
I20240503 14:12:07.146759 28600 tablet_bootstrap.cc:654] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.151283 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8410e8b6b511403ea609b1353855937b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.153496 28333 tablet_service.cc:1467] Processing CreateTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 (DEFAULT_TABLE table=default.Table_0c1dc7a948b74ac8808b2e85171a0996 [id=a6a9ad0f171448c2a3d3401e02ec62c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.155769 28321 rpcz_store.cc:276] 0503 14:12:05.838378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.839753 (+ 1375us) service_pool.cc:225] Handling call
0503 14:12:05.864033 (+ 24280us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:05.864062 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:06.462155 (+598093us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.123176 (+661021us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:07.125456 (+ 2280us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 14:12:07.156117 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 605531ed29214b1ca27011c5f5e96b11 consensus state has no leader
I20240503 14:12:07.113459 29119 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:07.159899 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b6c998e518541248a87e4c1551ec080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.159756 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.160177 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet c5297664a4d943c28adfb6ac4f061680 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.160157 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 351d5bfb12c34c5f9de90e2df1b36fc1 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:07.159848 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.167152 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet e135b60660f448a7974520f2f87cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.167212 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 64bae128469f4e8bb2d743320731446a (DEFAULT_TABLE table=default.Table_fc86c8af96ac4a70906e1a764f52c59b [id=cb2015955266483fa60f3fc692731e5f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:07.159945 28150 leader_election.cc:341] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.160012 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:07.168407 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 52233.6us
I20240503 14:12:07.168745 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4dc732bcb44222b4b15e5d77defe33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.169873 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 2221218847554cbd9d3242a9a6431e46 (DEFAULT_TABLE table=default.Table_4929b450373c4b689562f4a1d9e4b4f2 [id=7b85025507014faa9ce25ba44310b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.169239 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet e355aafb2f5b4337bedaac6fe5539dc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 80cd09e769be4bee95bbef9c3c550451) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:07.174597 28150 leader_election.cc:341] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.175061 28290 leader_election.cc:302] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:07.168038 29120 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:07.182668 29107 raft_consensus.cc:2781] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.184844 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_13c7175b4f704263a322c205dfb42854"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.185724 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13c7175b4f704263a322c205dfb42854 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.187995 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4d3ef3849e8a4cc3a225bdfea8c88884" } new_table_name: "default.Table_ede6e833710649e08e31516f0890c2d6" modify_external_catalogs: true
I20240503 14:12:07.189936 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_82fd7fffb051442b8351dccbec8d56ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.190752 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82fd7fffb051442b8351dccbec8d56ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.193187 28603 ts_tablet_manager.cc:1888] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:07.193101 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6d7202481c4747ddb44a78ba8bb0063f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.193804 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d7202481c4747ddb44a78ba8bb0063f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.197942 28150 leader_election.cc:302] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:07.177809 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19787034c24ebe929601d49cb21b3b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:07.198374 28216 raft_consensus.cc:2454] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:07.199330 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bae128469f4e8bb2d743320731446a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.200902 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.201242 29101 raft_consensus.cc:2726] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:07.201925 28150 consensus_peers.cc:485] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:07.201973 29107 raft_consensus.cc:482] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.202142 29107 raft_consensus.cc:3037] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.203675 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e135732daa240d7b537d589e0256892" } new_table_name: "default.Table_955617b294c94c299632de62b5973057" modify_external_catalogs: true
I20240503 14:12:07.205341 29101 consensus_queue.cc:1035] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.207654 28317 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58036 (request call id 47) took 1286 ms. Trace:
I20240503 14:12:07.207713 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d8cb0703edfb4e96ba081bd6f299d867"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.208493 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8cb0703edfb4e96ba081bd6f299d867 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.211381 29101 consensus_queue.cc:1035] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.212595 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2221218847554cbd9d3242a9a6431e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.215848 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_73db66f21afd468d887601ce8148489a" } new_table_name: "default.Table_6c623db75c5548ebb88cdc9a786d0555" modify_external_catalogs: true
I20240503 14:12:07.222947 28603 log.cc:1196] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7de7832122804eada175f710cc5bab5e
I20240503 14:12:07.225338 28614 tablet_bootstrap.cc:654] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:07.234535 28427 leader_election.cc:341] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.237069 28427 leader_election.cc:302] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:07.238008 29030 raft_consensus.cc:482] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.246292 29119 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58172 (request call id 218) took 1083 ms. Trace:
W20240503 14:12:07.246690 28150 consensus_peers.cc:485] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:07.223881 28317 rpcz_store.cc:276] 0503 14:12:05.892057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.894218 (+ 2161us) service_pool.cc:225] Handling call
0503 14:12:05.914480 (+ 20262us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x61800000fd18. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52be96619d 00007f52be965dbe 00007f52d1e1f62d 00007f52d1e1f5cc 00007f52cdf51246 00007f52cdf5199b 00007f52cdec06f5 00007f52c884c8e6 00007f52c884c615 00007f52c567804e
0503 14:12:06.022411 (+107931us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x6160000d3954. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52ca1aeb00 00007f52cdec0b7f 00007f52c884c8e6 00007f52c884c615 00007f52c567804e 00007f52c5676f2e 00007f52c567d405 00007f52c567dd2f 00007f52c567db58 0000000000a60bcc
0503 14:12:07.178343 (+1155932us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75233280}
I20240503 14:12:07.251633 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.251719 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet e135b60660f448a7974520f2f87cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.252508 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.257700 29134 tablet.cc:1722] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:07.261101 28028 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_eef00a50c2ce4752ab195301f31eddef [id=acb43ab939ff46428a795f618c6750dd]). Expected version 3 got 0
I20240503 14:12:07.265933 28653 ts_tablet_manager.cc:1888] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:07.266562 28653 log.cc:1196] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ab05178dc081480ba409aab7f143f849
I20240503 14:12:07.267247 28653 ts_tablet_manager.cc:1909] T ab05178dc081480ba409aab7f143f849 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:07.269626 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d6b50aed01a4bf8af8889036dd83606 consensus state has no leader
W20240503 14:12:07.274961 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7218cefb13ea4983b2111ed5fd0c997b consensus state has no leader
I20240503 14:12:07.275435 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d77014c66624f958b38f7cc83644891 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:07.279273 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dda39f0323ec47799d3d44c96f3ba605 consensus state has no leader
I20240503 14:12:07.279332 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6ee87d5c5b1b4bde9f21a38ab19981d7" } new_table_name: "default.Table_4ec913f6e2544dd98b3a26d8efffcc2f" modify_external_catalogs: true
I20240503 14:12:07.282042 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ba1e59290eb64bd488b68e68cb9b2548"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.282831 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba1e59290eb64bd488b68e68cb9b2548 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.283468 29134 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:07.284308 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_78f62674ace541fabf9047f21a7cadbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.284955 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78f62674ace541fabf9047f21a7cadbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.287813 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_eef00a50c2ce4752ab195301f31eddef [id=acb43ab939ff46428a795f618c6750dd]). Expected version 3 got 2
I20240503 14:12:07.288220 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8c7d57fc4486457e9848daad79f04356 (table default.Table_67ce7228ef934797ba768f947d2cb0aa [id=5cf73f88a73a4cc5966467ecfff2140f]). Expected version 1 got 0
I20240503 14:12:07.290016 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d121667544004d9e83e4daf3f58f7473"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.290752 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d121667544004d9e83e4daf3f58f7473 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.302675 28188 tablet_service.cc:1467] Processing CreateTablet for tablet ff4d28d3c2ea472191ad21355248a2fc (DEFAULT_TABLE table=default.Table_7b9e454770964e4594e5d043b06c2e78 [id=d4a1074afcca45b287b8d4416878f316]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.303260 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4d28d3c2ea472191ad21355248a2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.304064 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet c5297664a4d943c28adfb6ac4f061680 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.304210 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 120) took 4288 ms. Trace:
I20240503 14:12:07.304898 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 (DEFAULT_TABLE table=default.Table_cc68335896e1489789a7ca2101fa68ec [id=119495cd0b64414690f0d19bcce2ed78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.305084 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6202333af943cbbc0d09553c1c193c (DEFAULT_TABLE table=default.Table_fe566740b43c42658f0a0ff2de59038a [id=e9c7ec14573b44b79e70dc70fc9226c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.305385 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4084bd4fbf604042bb33ff03c3f8cc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.305961 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 3c61b9232ebd4431a3d4202ad971d540 (DEFAULT_TABLE table=default.Table_34ff5e18a1334c7faa054816f0837ee2 [id=e5156a3dcf274657af2bb43bcaabfea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.306854 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6202333af943cbbc0d09553c1c193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.239097 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19787034c24ebe929601d49cb21b3b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:07.310503 29030 raft_consensus.cc:504] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.311296 29030 leader_election.cc:288] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:07.311553 29030 raft_consensus.cc:2726] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:07.311735 29030 raft_consensus.cc:482] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.311906 29030 raft_consensus.cc:504] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.312719 29030 leader_election.cc:288] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:07.312855 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c61b9232ebd4431a3d4202ad971d540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.315950 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 (DEFAULT_TABLE table=default.Table_4d3ef3849e8a4cc3a225bdfea8c88884 [id=3d49172b73ba44538840b651a85b24e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.316558 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6b50aed01a4bf8af8889036dd83606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.317384 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9953322d44f34ce5ad1123ba2784b66f" } modify_external_catalogs: true
I20240503 14:12:07.317660 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9953322d44f34ce5ad1123ba2784b66f" } modify_external_catalogs: true
I20240503 14:12:07.318464 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet e135b60660f448a7974520f2f87cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.319473 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dbb3e0a033c84560a6458571881b25e5" } modify_external_catalogs: true
I20240503 14:12:07.319746 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dbb3e0a033c84560a6458571881b25e5" } modify_external_catalogs: true
I20240503 14:12:07.320464 28661 ts_tablet_manager.cc:1909] T 7de7832122804eada175f710cc5bab5e P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:07.323956 28603 ts_tablet_manager.cc:1909] T 7de7832122804eada175f710cc5bab5e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:07.324577 28335 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58028 (request call id 237) took 1005 ms. Trace:
I20240503 14:12:07.326404 28324 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58028 (request call id 235) took 1065 ms. Trace:
I20240503 14:12:07.327536 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 145) took 3411 ms. Trace:
I20240503 14:12:07.327700 28661 rpcz_store.cc:276] 0503 14:12:03.916008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.925108 (+ 9100us) service_pool.cc:225] Handling call
0503 14:12:07.051553 (+3126445us) ts_tablet_manager.cc:1120] Deleting tablet 7de7832122804eada175f710cc5bab5e
0503 14:12:07.059262 (+ 7709us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.070963 (+ 11701us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.138340 (+ 67377us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.327418 (+189078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2719132}
I20240503 14:12:07.328260 28661 tablet_replica.cc:330] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:07.328593 28661 raft_consensus.cc:2229] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.328528 28335 rpcz_store.cc:276] 0503 14:12:06.247894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.509705 (+261811us) service_pool.cc:225] Handling call
0503 14:12:06.543787 (+ 34082us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:06.543823 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:07.019965 (+476142us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.250486 (+230521us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:07.253527 (+ 3041us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 14:12:07.328881 28322 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58028 (request call id 236) took 1036 ms. Trace:
I20240503 14:12:07.329051 28661 raft_consensus.cc:2258] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.329602 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 351d5bfb12c34c5f9de90e2df1b36fc1
I20240503 14:12:07.329763 28661 ts_tablet_manager.cc:1875] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.330174 28027 catalog_manager.cc:6554] ReplaceTablet: tablet 351d5bfb12c34c5f9de90e2df1b36fc1 deleted and replaced by tablet d6f593467bf4482cb926b63480790ad4
I20240503 14:12:07.332500 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 74df7781cc6a4dd6b6fd78a728ffd71b
I20240503 14:12:07.310122 28653 rpcz_store.cc:276] 0503 14:12:02.979910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.979972 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:07.030527 (+4050555us) ts_tablet_manager.cc:1120] Deleting tablet ab05178dc081480ba409aab7f143f849
0503 14:12:07.030538 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.152185 (+121647us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.265793 (+113608us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.268192 (+ 2399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4050203}
I20240503 14:12:07.333048 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 74df7781cc6a4dd6b6fd78a728ffd71b deleted and replaced by tablet 4e8411f474c247c2abfd0bd9682fb8a2
I20240503 14:12:07.333446 28653 tablet_replica.cc:330] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:07.333680 28653 raft_consensus.cc:2229] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.333920 28653 raft_consensus.cc:2258] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.334786 28653 ts_tablet_manager.cc:1875] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.336052 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet e355aafb2f5b4337bedaac6fe5539dc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 80cd09e769be4bee95bbef9c3c550451) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.340620 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1d77014c66624f958b38f7cc83644891
I20240503 14:12:07.341190 28600 tablet_bootstrap.cc:492] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:07.341274 28026 catalog_manager.cc:6554] ReplaceTablet: tablet 1d77014c66624f958b38f7cc83644891 deleted and replaced by tablet 20b176a071ba4f3fbff2a64d6e46027a
I20240503 14:12:07.341408 28600 ts_tablet_manager.cc:1367] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.479s user 0.008s sys 0.005s
I20240503 14:12:07.338316 28290 leader_election.cc:302] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
W20240503 14:12:07.342651 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77f0081cc2cb48aaa368741109032fd1 consensus state has no leader
I20240503 14:12:07.342685 28600 raft_consensus.cc:348] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.342986 28600 raft_consensus.cc:374] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.343130 28600 raft_consensus.cc:729] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.343418 28600 consensus_queue.cc:260] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.347957 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7d8f2c48beb5472fb05f0cbd475f7182" } modify_external_catalogs: true
I20240503 14:12:07.348299 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7d8f2c48beb5472fb05f0cbd475f7182" } modify_external_catalogs: true
I20240503 14:12:07.350129 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_355c3b14ae4f480dba47c2c90dc9dba2" } modify_external_catalogs: true
I20240503 14:12:07.350368 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_355c3b14ae4f480dba47c2c90dc9dba2" } modify_external_catalogs: true
I20240503 14:12:07.354234 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]). Expected version 2 got 0
I20240503 14:12:07.354588 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_eef00a50c2ce4752ab195301f31eddef [id=acb43ab939ff46428a795f618c6750dd]). Expected version 3 got 0
I20240503 14:12:07.357090 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_de96804c306a4b0e91a2e6bffd5943a8" } modify_external_catalogs: true
I20240503 14:12:07.357466 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_de96804c306a4b0e91a2e6bffd5943a8" } modify_external_catalogs: true
I20240503 14:12:07.360433 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_17677d89d0664bb4b835cd1c24a9f27c" } new_table_name: "default.Table_6595add4a286443f9cfe220b536c5709" modify_external_catalogs: true
I20240503 14:12:07.362802 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6932067dd9e648e0a368cea1aae2eaa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:07.363404 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 351d5bfb12c34c5f9de90e2df1b36fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet d6f593467bf4482cb926b63480790ad4) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:07.363497 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6932067dd9e648e0a368cea1aae2eaa3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.341221 28324 rpcz_store.cc:276] 0503 14:12:06.243593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.457091 (+213498us) service_pool.cc:225] Handling call
0503 14:12:06.501673 (+ 44582us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:06.501715 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:06.970459 (+468744us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.303812 (+333353us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:07.308584 (+ 4772us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 14:12:07.338451 28322 rpcz_store.cc:276] 0503 14:12:06.243723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.488053 (+244330us) service_pool.cc:225] Handling call
0503 14:12:06.537694 (+ 49641us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:06.537731 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:07.144918 (+607187us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.279576 (+134658us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:07.279873 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 14:12:07.344828 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 149) took 3534 ms. Trace:
I20240503 14:12:07.364552 28603 rpcz_store.cc:276] 0503 14:12:03.790101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.790482 (+ 381us) service_pool.cc:225] Handling call
0503 14:12:06.278021 (+2487539us) ts_tablet_manager.cc:1120] Deleting tablet 7de7832122804eada175f710cc5bab5e
0503 14:12:06.278036 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.012388 (+734352us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.193019 (+180631us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.324453 (+131434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2474039}
I20240503 14:12:07.365168 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 150) took 3571 ms. Trace:
I20240503 14:12:07.365289 28603 rpcz_store.cc:276] 0503 14:12:03.794004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.794915 (+ 911us) service_pool.cc:225] Handling call
0503 14:12:07.364912 (+3569997us) ts_tablet_manager.cc:1120] Deleting tablet 93462d2b1a474bd688d6266cd3b0695c
0503 14:12:07.364925 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.365072 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3560534}
I20240503 14:12:07.365697 28603 tablet_replica.cc:330] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:07.365934 28603 raft_consensus.cc:2229] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.351192 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 3c61b9232ebd4431a3d4202ad971d540 (DEFAULT_TABLE table=default.Table_34ff5e18a1334c7faa054816f0837ee2 [id=e5156a3dcf274657af2bb43bcaabfea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.367136 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c61b9232ebd4431a3d4202ad971d540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.352527 28325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58028 (request call id 233) took 1091 ms. Trace:
I20240503 14:12:07.367746 28325 rpcz_store.cc:276] 0503 14:12:06.241609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.413346 (+171737us) service_pool.cc:225] Handling call
0503 14:12:06.526544 (+113198us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:06.526607 (+ 63us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:07.209765 (+683158us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.332408 (+122643us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:07.332631 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 14:12:07.355866 29107 raft_consensus.cc:504] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.369017 29107 leader_election.cc:288] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:07.369443 29107 raft_consensus.cc:2781] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.369562 29107 raft_consensus.cc:482] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.369707 29107 raft_consensus.cc:3037] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.359982 28326 tablet_service.cc:1467] Processing CreateTablet for tablet ff4d28d3c2ea472191ad21355248a2fc (DEFAULT_TABLE table=default.Table_7b9e454770964e4594e5d043b06c2e78 [id=d4a1074afcca45b287b8d4416878f316]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.370924 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4d28d3c2ea472191ad21355248a2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.371044 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ab05178dc081480ba409aab7f143f849 (table default.Table_adb8e4f58ba042f0a57523a25228ca92 [id=37c31c087c65469f96f4322d84ba161f]) successfully deleted
I20240503 14:12:07.340966 29120 tablet.cc:1722] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:07.376579 28024 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_afc737ad630d4de6b239f807771fa6a1 [id=68a4a475afc24b95b5388e43deb4632b] alter complete (version 1)
I20240503 14:12:07.377457 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f010ec576f0a40cb91c71b23da613670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.378165 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f010ec576f0a40cb91c71b23da613670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.384889 29142 tablet.cc:1722] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:07.386931 28604 tablet_bootstrap.cc:492] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:07.387259 28604 ts_tablet_manager.cc:1367] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.342s user 0.004s sys 0.011s
I20240503 14:12:07.388913 28604 raft_consensus.cc:348] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.301769 29119 rpcz_store.cc:276] 0503 14:12:06.160956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.161020 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:07.244826 (+1083806us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:06.162084 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:06.174232 (+ 12148us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:06.174243 (+ 11us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:06.174250 (+ 7us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:06.174293 (+ 43us) alter_schema_op.cc:119] START. Timestamp: 1714745526174216
0503 14:12:06.174297 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:06.174640 (+ 343us) log.cc:842] Serialized 173 byte log entry
0503 14:12:06.957659 (+783019us) op_driver.cc:444] REPLICATION: finished
0503 14:12:07.084836 (+127177us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:07.160599 (+ 75763us) log.cc:842] Serialized 5 byte log entry
0503 14:12:07.232416 (+ 71817us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.244350 (+ 11934us) log.cc:842] Serialized 14 byte log entry
0503 14:12:07.244776 (+ 426us) alter_schema_op.cc:65] Released schema lock
0503 14:12:07.244781 (+ 5us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":127032,"prepare.queue_time_us":529,"prepare.run_cpu_time_us":14254,"prepare.run_wall_time_us":41527,"replication_time_us":783327,"thread_start_us":169495,"threads_started":3,"wal-append.queue_time_us":28964}]]}
I20240503 14:12:07.389513 28604 raft_consensus.cc:374] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.389662 28604 raft_consensus.cc:729] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.390022 28604 consensus_queue.cc:260] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.391896 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d77014c66624f958b38f7cc83644891 with delete_type TABLET_DATA_DELETED (replaced by tablet 20b176a071ba4f3fbff2a64d6e46027a) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.393075 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8411f474c247c2abfd0bd9682fb8a2) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.373751 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 (DEFAULT_TABLE table=default.Table_cc68335896e1489789a7ca2101fa68ec [id=119495cd0b64414690f0d19bcce2ed78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.366495 28603 raft_consensus.cc:2258] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.376814 29107 raft_consensus.cc:504] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:07.394316 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 161643us
I20240503 14:12:07.396713 29139 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:07.396977 29119 tablet.cc:1722] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:07.397248 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet e355aafb2f5b4337bedaac6fe5539dc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 80cd09e769be4bee95bbef9c3c550451) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.398149 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet eee989ff083041dc8dcd2cd1b220780d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.400401 28323 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58028 (request call id 218) took 1241 ms. Trace:
I20240503 14:12:07.401623 28600 ts_tablet_manager.cc:1398] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.031s user 0.001s sys 0.002s
I20240503 14:12:07.403867 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 3c61b9232ebd4431a3d4202ad971d540 (DEFAULT_TABLE table=default.Table_34ff5e18a1334c7faa054816f0837ee2 [id=e5156a3dcf274657af2bb43bcaabfea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.404433 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c61b9232ebd4431a3d4202ad971d540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.408684 28600 tablet_bootstrap.cc:492] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:07.412895 28323 rpcz_store.cc:276] 0503 14:12:06.094322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.094600 (+ 278us) service_pool.cc:225] Handling call
0503 14:12:06.170614 (+ 76014us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:06.170638 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:07.105583 (+934945us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.335700 (+230117us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:07.335881 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 14:12:07.413375 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 (DEFAULT_TABLE table=default.Table_4d3ef3849e8a4cc3a225bdfea8c88884 [id=3d49172b73ba44538840b651a85b24e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.414064 28603 ts_tablet_manager.cc:1875] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.416064 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4084bd4fbf604042bb33ff03c3f8cc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.419353 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19787034c24ebe929601d49cb21b3b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:07.419461 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19787034c24ebe929601d49cb21b3b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:07.419631 28217 raft_consensus.cc:3037] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.419771 28493 raft_consensus.cc:3037] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.422957 28493 raft_consensus.cc:2454] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:07.423489 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee989ff083041dc8dcd2cd1b220780d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:07.423823 28212 raft_consensus.cc:3037] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.425844 28217 raft_consensus.cc:2454] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:07.426267 29142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51314 (request call id 229) took 1033 ms. Trace:
I20240503 14:12:07.426406 29142 rpcz_store.cc:276] 0503 14:12:06.392679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.392735 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:07.426213 (+1033478us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:06.625334 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:06.625482 (+ 148us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:06.625501 (+ 19us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:06.625509 (+ 8us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:06.625564 (+ 55us) alter_schema_op.cc:119] START. Timestamp: 1714745526625527
0503 14:12:06.625568 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:06.625934 (+ 366us) log.cc:842] Serialized 173 byte log entry
0503 14:12:07.161214 (+535280us) op_driver.cc:444] REPLICATION: finished
0503 14:12:07.384776 (+223562us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:07.385558 (+ 782us) log.cc:842] Serialized 5 byte log entry
0503 14:12:07.425755 (+ 40197us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.426128 (+ 373us) log.cc:842] Serialized 14 byte log entry
0503 14:12:07.426177 (+ 49us) alter_schema_op.cc:65] Released schema lock
0503 14:12:07.426181 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":223453,"prepare.queue_time_us":220571,"prepare.run_cpu_time_us":2297,"prepare.run_wall_time_us":148925,"replication_time_us":535615,"thread_start_us":747169,"threads_started":3,"wal-append.queue_time_us":147047}]]}
I20240503 14:12:07.427093 28212 raft_consensus.cc:2454] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:07.427294 29142 tablet.cc:1722] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:07.427975 29107 leader_election.cc:288] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:07.428602 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1321728b41b14e7898208cc20c6d1f08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.429430 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1321728b41b14e7898208cc20c6d1f08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.419752 29135 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:07.431381 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee989ff083041dc8dcd2cd1b220780d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:07.431790 28489 raft_consensus.cc:3037] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.432133 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6b50aed01a4bf8af8889036dd83606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.435111 28489 raft_consensus.cc:2454] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:07.436183 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 (DEFAULT_TABLE table=default.Table_cc68335896e1489789a7ca2101fa68ec [id=119495cd0b64414690f0d19bcce2ed78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.436841 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4084bd4fbf604042bb33ff03c3f8cc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.437825 28604 ts_tablet_manager.cc:1398] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20240503 14:12:07.438357 28604 tablet_bootstrap.cc:492] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:07.439982 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6202333af943cbbc0d09553c1c193c (DEFAULT_TABLE table=default.Table_fe566740b43c42658f0a0ff2de59038a [id=e9c7ec14573b44b79e70dc70fc9226c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.440888 28474 tablet_service.cc:1467] Processing CreateTablet for tablet ff4d28d3c2ea472191ad21355248a2fc (DEFAULT_TABLE table=default.Table_7b9e454770964e4594e5d043b06c2e78 [id=d4a1074afcca45b287b8d4416878f316]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.441426 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4d28d3c2ea472191ad21355248a2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.441923 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 (DEFAULT_TABLE table=default.Table_4d3ef3849e8a4cc3a225bdfea8c88884 [id=3d49172b73ba44538840b651a85b24e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.442356 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6b50aed01a4bf8af8889036dd83606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.448436 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6202333af943cbbc0d09553c1c193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.452936 28604 tablet_bootstrap.cc:654] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.463424 28600 tablet_bootstrap.cc:654] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.465386 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6202333af943cbbc0d09553c1c193c (DEFAULT_TABLE table=default.Table_fe566740b43c42658f0a0ff2de59038a [id=e9c7ec14573b44b79e70dc70fc9226c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.465785 29134 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58172 (request call id 223) took 1241 ms. Trace:
I20240503 14:12:07.465989 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6202333af943cbbc0d09553c1c193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.466609 28604 tablet_bootstrap.cc:492] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:07.467003 28604 ts_tablet_manager.cc:1367] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.006s
I20240503 14:12:07.468914 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8411f474c247c2abfd0bd9682fb8a2) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:07.468905 28290 leader_election.cc:341] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.467638 29134 rpcz_store.cc:276] 0503 14:12:06.206754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.207488 (+ 734us) service_pool.cc:225] Handling call
0503 14:12:07.445347 (+1237859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:06.208791 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:06.208894 (+ 103us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:07.246863 (+1037969us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:07.246879 (+ 16us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:07.246949 (+ 70us) alter_schema_op.cc:119] START. Timestamp: 1714745527246913
0503 14:12:07.246953 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:07.247233 (+ 280us) log.cc:842] Serialized 173 byte log entry
0503 14:12:07.282865 (+ 35632us) op_driver.cc:444] REPLICATION: finished
0503 14:12:07.283390 (+ 525us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:07.308038 (+ 24648us) log.cc:842] Serialized 5 byte log entry
0503 14:12:07.411866 (+103828us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.444463 (+ 32597us) log.cc:842] Serialized 14 byte log entry
0503 14:12:07.445143 (+ 680us) alter_schema_op.cc:65] Released schema lock
0503 14:12:07.445150 (+ 7us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":461,"prepare.queue_time_us":87,"prepare.run_cpu_time_us":86822,"prepare.run_wall_time_us":1039017,"raft.queue_time_us":48,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":353,"replication_time_us":35883,"spinlock_wait_cycles":20992}]]}
I20240503 14:12:07.484123 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 351d5bfb12c34c5f9de90e2df1b36fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet d6f593467bf4482cb926b63480790ad4) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.484515 28604 raft_consensus.cc:348] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.485018 28604 raft_consensus.cc:374] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.485225 28604 raft_consensus.cc:729] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.486191 28604 consensus_queue.cc:260] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.487324 28604 ts_tablet_manager.cc:1398] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.020s user 0.000s sys 0.003s
I20240503 14:12:07.488626 28604 tablet_bootstrap.cc:492] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:07.488063 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d77014c66624f958b38f7cc83644891 with delete_type TABLET_DATA_DELETED (replaced by tablet 20b176a071ba4f3fbff2a64d6e46027a) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.487534 28661 ts_tablet_manager.cc:1888] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:07.489699 28661 log.cc:1196] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/b166fe6366ba496a869af1b106fe9d8e
I20240503 14:12:07.490357 28661 ts_tablet_manager.cc:1909] T b166fe6366ba496a869af1b106fe9d8e P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:07.494947 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 147) took 3568 ms. Trace:
I20240503 14:12:07.498245 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1693dd36a3d84d88a49690c537590083" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:07.498976 28661 rpcz_store.cc:276] 0503 14:12:03.926794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.949668 (+ 22874us) service_pool.cc:225] Handling call
0503 14:12:07.328036 (+3378368us) ts_tablet_manager.cc:1120] Deleting tablet b166fe6366ba496a869af1b106fe9d8e
0503 14:12:07.328050 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.339072 (+ 11022us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.469276 (+130204us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.494874 (+ 25598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2995564}
I20240503 14:12:07.499572 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 144) took 3605 ms. Trace:
W20240503 14:12:07.499722 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 47384.7us
I20240503 14:12:07.499738 28661 rpcz_store.cc:276] 0503 14:12:03.913410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.925043 (+ 11633us) service_pool.cc:225] Handling call
0503 14:12:07.499350 (+3574307us) ts_tablet_manager.cc:1120] Deleting tablet a8d2600031df4d77b9afe636a9b5ad40
0503 14:12:07.499361 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.499514 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3161955}
I20240503 14:12:07.496284 29139 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:07.497503 29144 tablet.cc:1722] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:07.502008 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7de7832122804eada175f710cc5bab5e (table default.Table_cc4fe0dac143484396ef862744cc9967 [id=e324426e1ff447a38c25b3867724fc4c]) successfully deleted
I20240503 14:12:07.504647 28214 raft_consensus.cc:2454] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:07.505084 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee989ff083041dc8dcd2cd1b220780d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:07.506023 28604 tablet_bootstrap.cc:654] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.503798 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 169) took 3048 ms. Trace:
I20240503 14:12:07.508944 28212 raft_consensus.cc:2379] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
W20240503 14:12:07.509256 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 93462d2b1a474bd688d6266cd3b0695c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93462d2b1a474bd688d6266cd3b0695c already in progress: creating tablet
I20240503 14:12:07.509898 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet 351d5bfb12c34c5f9de90e2df1b36fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet d6f593467bf4482cb926b63480790ad4) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.510748 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7de7832122804eada175f710cc5bab5e (table default.Table_cc4fe0dac143484396ef862744cc9967 [id=e324426e1ff447a38c25b3867724fc4c]) successfully deleted
I20240503 14:12:07.511425 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_17a1019962da48c0992b5452d4a67aae [id=548632b6d08e4435ad561b074e5e5b14]) successfully deleted
W20240503 14:12:07.511693 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a8d2600031df4d77b9afe636a9b5ad40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8d2600031df4d77b9afe636a9b5ad40 already in progress: creating tablet
I20240503 14:12:07.516400 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2404807f8500437ababca53f6388b636
I20240503 14:12:07.518698 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8411f474c247c2abfd0bd9682fb8a2) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.518986 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d77014c66624f958b38f7cc83644891 with delete_type TABLET_DATA_DELETED (replaced by tablet 20b176a071ba4f3fbff2a64d6e46027a) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.519795 28653 ts_tablet_manager.cc:1888] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:07.519855 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6371e68ee85a47cdb0e8e21f7124d94a
I20240503 14:12:07.519996 28653 log.cc:1196] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/bacf2d1091ee458c961c691093ea65a9
W20240503 14:12:07.520437 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65853212c2404597b015ddc8ea99ee47 consensus state has no leader
I20240503 14:12:07.520936 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_10446f75b72747c0bfa9aa56408f9cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.521584 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10446f75b72747c0bfa9aa56408f9cd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.523119 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6dab0296a0894a63afc16ed466c591d5
I20240503 14:12:07.523942 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_00017e9cb2c6403f8a862b8cb661979c" } new_table_name: "default.Table_38aab413057643f78d942ab06e5905f7" modify_external_catalogs: true
I20240503 14:12:07.525816 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_30902594a17047d0ac5f2784f7a1d2eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.526472 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30902594a17047d0ac5f2784f7a1d2eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.527971 28600 tablet_bootstrap.cc:492] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:07.528353 28653 ts_tablet_manager.cc:1909] T bacf2d1091ee458c961c691093ea65a9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:07.528867 28614 tablet_bootstrap.cc:492] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:07.529284 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 119) took 4549 ms. Trace:
I20240503 14:12:07.529438 28653 rpcz_store.cc:276] 0503 14:12:02.979711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:02.980027 (+ 316us) service_pool.cc:225] Handling call
0503 14:12:07.333316 (+4353289us) ts_tablet_manager.cc:1120] Deleting tablet bacf2d1091ee458c961c691093ea65a9
0503 14:12:07.333325 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.376452 (+ 43127us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.466370 (+ 89918us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.529233 (+ 62863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4352975}
I20240503 14:12:07.529811 28653 tablet_replica.cc:330] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:07.530050 28653 raft_consensus.cc:2229] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.530349 28653 pending_rounds.cc:62] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:07.530447 28653 pending_rounds.cc:68] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597672362061824 op_type: NO_OP noop_request { }
I20240503 14:12:07.530604 28653 raft_consensus.cc:2866] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:07.530747 28653 raft_consensus.cc:2258] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.531518 28653 ts_tablet_manager.cc:1875] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.532116 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:07.534512 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1693dd36a3d84d88a49690c537590083" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:07.536279 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_55ff9f3cd6f84368add48d9cecc3fb1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.537055 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55ff9f3cd6f84368add48d9cecc3fb1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.539584 29119 tablet.cc:1722] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:07.541352 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee989ff083041dc8dcd2cd1b220780d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:07.541326 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_eef00a50c2ce4752ab195301f31eddef" } new_table_name: "default.Table_573a1680a2fa4db793e25023dd9eea51" modify_external_catalogs: true
I20240503 14:12:07.542305 28600 ts_tablet_manager.cc:1367] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.003s
I20240503 14:12:07.542492 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_35f666e8e6c54a82893c615d5ffd476d" } new_table_name: "default.Table_4c09a9ca5e3d4e01820ece646f161b1c" modify_external_catalogs: true
I20240503 14:12:07.543504 28600 raft_consensus.cc:348] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.543807 28600 raft_consensus.cc:374] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.543913 28600 raft_consensus.cc:729] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.548763 28614 ts_tablet_manager.cc:1367] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.463s user 0.013s sys 0.010s
I20240503 14:12:07.549836 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 2404807f8500437ababca53f6388b636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.550803 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dab0296a0894a63afc16ed466c591d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.551728 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.553330 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_11703c3db99a47cc85741ba32923f695"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.554103 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11703c3db99a47cc85741ba32923f695 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.556294 28600 consensus_queue.cc:260] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.556566 28354 raft_consensus.cc:2379] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:07.557091 28600 ts_tablet_manager.cc:1398] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:12:07.560645 28314 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58036 (request call id 49) took 1560 ms. Trace:
I20240503 14:12:07.560830 28314 rpcz_store.cc:276] 0503 14:12:05.999605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.999675 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:07.228245 (+1228570us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x6180000a108c. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52cb6bc12e 00007f52cb4c279f 00007f52cdec0988 00007f52c884c8e6 00007f52c884c615 00007f52c567804e 00007f52c5676f2e 00007f52c567d405 00007f52c567dd2f 00007f52c567db58
0503 14:12:07.556968 (+328723us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x6160000b2c54. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52ca1aeb00 00007f52cdec0b7f 00007f52c884c8e6 00007f52c884c615 00007f52c567804e 00007f52c5676f2e 00007f52c567d405 00007f52c567dd2f 00007f52c567db58 0000000000a60bcc
0503 14:12:07.559838 (+ 2870us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":597138048}
I20240503 14:12:07.561352 28653 ts_tablet_manager.cc:1888] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:07.561547 28653 log.cc:1196] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/9dbabe5beb9a42c69cb64e1b364d75a1
I20240503 14:12:07.562223 28653 ts_tablet_manager.cc:1909] T 9dbabe5beb9a42c69cb64e1b364d75a1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:07.563781 28600 tablet_bootstrap.cc:492] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:07.570014 28600 tablet_bootstrap.cc:654] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.506742 28427 leader_election.cc:302] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:07.576599 29144 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51314 (request call id 231) took 1164 ms. Trace:
I20240503 14:12:07.577109 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 121) took 4571 ms. Trace:
I20240503 14:12:07.577355 28653 rpcz_store.cc:276] 0503 14:12:03.005374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.005436 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:07.529724 (+4524288us) ts_tablet_manager.cc:1120] Deleting tablet 9dbabe5beb9a42c69cb64e1b364d75a1
0503 14:12:07.529734 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.546566 (+ 16832us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.561143 (+ 14577us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.577022 (+ 15879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4523888}
W20240503 14:12:07.578296 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 64261.1us
I20240503 14:12:07.578521 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 132) took 4454 ms. Trace:
I20240503 14:12:07.578750 28653 rpcz_store.cc:276] 0503 14:12:03.124373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.124562 (+ 189us) service_pool.cc:225] Handling call
0503 14:12:07.578241 (+4453679us) ts_tablet_manager.cc:1120] Deleting tablet 40da5f5fff94462d8af740f3ee337e76
0503 14:12:07.578252 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.578390 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4453330}
I20240503 14:12:07.579924 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 133) took 4232 ms. Trace:
I20240503 14:12:07.566915 28290 leader_election.cc:302] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:07.589936 29107 raft_consensus.cc:2781] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.590198 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dab0296a0894a63afc16ed466c591d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.580242 28653 rpcz_store.cc:276] 0503 14:12:03.347055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.348347 (+ 1292us) service_pool.cc:225] Handling call
0503 14:12:07.579281 (+4230934us) ts_tablet_manager.cc:1120] Deleting tablet cae4969fc9474fdbaeae1b4d09e4875b
0503 14:12:07.579293 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.579443 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4230561}
I20240503 14:12:07.593005 29107 raft_consensus.cc:686] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:07.593489 29107 consensus_queue.cc:237] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.594334 28653 tablet_replica.cc:330] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:07.594897 28653 raft_consensus.cc:2229] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.596155 28653 raft_consensus.cc:2258] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:07.597335 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 44418.6us
I20240503 14:12:07.597733 28653 ts_tablet_manager.cc:1875] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.599494 28290 leader_election.cc:302] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:07.599925 28956 raft_consensus.cc:2781] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.600639 29140 raft_consensus.cc:2781] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.600824 29140 raft_consensus.cc:482] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.601034 29140 raft_consensus.cc:3037] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.601476 28956 raft_consensus.cc:686] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:07.601816 28956 consensus_queue.cc:237] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.590641 28614 raft_consensus.cc:348] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.601641 29144 rpcz_store.cc:276] 0503 14:12:06.410277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.410374 (+ 97us) service_pool.cc:225] Handling call
0503 14:12:07.574969 (+1164595us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:06.723712 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:06.723942 (+ 230us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:06.723953 (+ 11us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:06.723960 (+ 7us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:06.724027 (+ 67us) alter_schema_op.cc:119] START. Timestamp: 1714745526723978
0503 14:12:06.724030 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:06.724536 (+ 506us) log.cc:842] Serialized 173 byte log entry
0503 14:12:07.497257 (+772721us) op_driver.cc:444] REPLICATION: finished
0503 14:12:07.497398 (+ 141us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:07.505340 (+ 7942us) log.cc:842] Serialized 5 byte log entry
0503 14:12:07.574462 (+ 69122us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.574866 (+ 404us) log.cc:842] Serialized 14 byte log entry
0503 14:12:07.574920 (+ 54us) alter_schema_op.cc:65] Released schema lock
0503 14:12:07.574925 (+ 5us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34,"prepare.queue_time_us":241686,"prepare.run_cpu_time_us":951,"prepare.run_wall_time_us":951,"raft.queue_time_us":203,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":308,"replication_time_us":773194,"thread_start_us":139809,"threads_started":1}]]}
I20240503 14:12:07.602782 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.603055 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 2404807f8500437ababca53f6388b636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:07.603391 28427 leader_election.cc:341] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.603559 28614 raft_consensus.cc:374] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.603698 28614 raft_consensus.cc:729] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.603861 28427 leader_election.cc:302] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:07.604622 28754 raft_consensus.cc:2726] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:07.604525 28614 consensus_queue.cc:260] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.605446 28614 ts_tablet_manager.cc:1398] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.056s user 0.001s sys 0.002s
I20240503 14:12:07.606118 29140 raft_consensus.cc:504] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.606716 28614 tablet_bootstrap.cc:492] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:07.607807 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1693dd36a3d84d88a49690c537590083" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:07.608143 28212 raft_consensus.cc:3037] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.611305 28600 tablet_bootstrap.cc:492] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:07.611516 28212 raft_consensus.cc:2454] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:07.611642 28600 ts_tablet_manager.cc:1367] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.000s
I20240503 14:12:07.612825 28427 leader_election.cc:302] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:07.613054 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.612964 28600 raft_consensus.cc:348] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.613615 29153 raft_consensus.cc:2781] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.613927 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.614094 29153 raft_consensus.cc:686] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:07.614229 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet bacf2d1091ee458c961c691093ea65a9 (table default.Table_71af49729f324730af4113012443a75c [id=50ac1df3e7c54c4ca543099d3019d337]) successfully deleted
I20240503 14:12:07.614537 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 9dbabe5beb9a42c69cb64e1b364d75a1 (table default.Table_6d7feb39cce84e66b75914a6bcbcfcc9 [id=56b86fbb2fff40cfa3729f88092d57ca]) successfully deleted
I20240503 14:12:07.614567 29153 consensus_queue.cc:237] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:07.614970 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 40da5f5fff94462d8af740f3ee337e76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40da5f5fff94462d8af740f3ee337e76 already in progress: creating tablet
W20240503 14:12:07.615271 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet cae4969fc9474fdbaeae1b4d09e4875b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cae4969fc9474fdbaeae1b4d09e4875b already in progress: creating tablet
I20240503 14:12:07.616014 29120 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58028 (request call id 216) took 1578 ms. Trace:
I20240503 14:12:07.616747 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 2404807f8500437ababca53f6388b636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.617014 29140 leader_election.cc:288] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:07.617015 28600 raft_consensus.cc:374] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.617306 28600 raft_consensus.cc:729] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.617314 28661 rpcz_store.cc:276] 0503 14:12:04.452234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.771694 (+319460us) service_pool.cc:225] Handling call
0503 14:12:07.499970 (+2728276us) ts_tablet_manager.cc:1120] Deleting tablet 779f373ef12a4da091309fcf247b5693
0503 14:12:07.499980 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.500172 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417556}
I20240503 14:12:07.617735 28600 consensus_queue.cc:260] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.616297 29120 rpcz_store.cc:276] 0503 14:12:06.027704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.027832 (+ 128us) service_pool.cc:225] Handling call
0503 14:12:07.606558 (+1578726us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:06.690235 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:06.697488 (+ 7253us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:06.697499 (+ 11us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:06.697508 (+ 9us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:06.697616 (+ 108us) alter_schema_op.cc:119] START. Timestamp: 1714745526697545
0503 14:12:06.697620 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:06.698469 (+ 849us) log.cc:842] Serialized 173 byte log entry
0503 14:12:07.176681 (+478212us) op_driver.cc:444] REPLICATION: finished
0503 14:12:07.326010 (+149329us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:07.393841 (+ 67831us) log.cc:842] Serialized 5 byte log entry
0503 14:12:07.606117 (+212276us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.606463 (+ 346us) log.cc:842] Serialized 14 byte log entry
0503 14:12:07.606522 (+ 59us) alter_schema_op.cc:65] Released schema lock
0503 14:12:07.606525 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149275,"prepare.queue_time_us":508480,"prepare.run_cpu_time_us":1034,"prepare.run_wall_time_us":17045,"raft.queue_time_us":59,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":253,"replication_time_us":479029,"thread_start_us":474628,"threads_started":2}]]}
I20240503 14:12:07.618779 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dab0296a0894a63afc16ed466c591d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.619202 28661 tablet_replica.cc:330] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:07.619104 28600 ts_tablet_manager.cc:1398] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:07.619876 28661 raft_consensus.cc:2229] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.620561 28661 raft_consensus.cc:2258] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.620790 28600 tablet_bootstrap.cc:492] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:07.623606 29113 raft_consensus.cc:482] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.623826 29113 raft_consensus.cc:504] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.624797 29113 leader_election.cc:288] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:07.625391 28661 ts_tablet_manager.cc:1875] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.626248 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989ea2f962a7492d8c32e4becec1457d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:07.626848 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989ea2f962a7492d8c32e4becec1457d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:07.627501 28150 leader_election.cc:341] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:07.627831 28150 leader_election.cc:341] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.628047 28150 leader_election.cc:302] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:07.628697 29113 raft_consensus.cc:2726] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:07.630636 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.636245 28604 tablet_bootstrap.cc:492] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:07.636620 28604 ts_tablet_manager.cc:1367] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.148s user 0.013s sys 0.001s
I20240503 14:12:07.637946 28604 raft_consensus.cc:348] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.638417 28604 raft_consensus.cc:374] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.639694 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1693dd36a3d84d88a49690c537590083" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:07.640246 28604 raft_consensus.cc:729] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.640770 28604 consensus_queue.cc:260] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.640972 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.641417 29120 tablet.cc:1722] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:07.641846 29135 tablet.cc:1722] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:07.644317 28653 ts_tablet_manager.cc:1888] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:07.645632 28653 log.cc:1196] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a8d2600031df4d77b9afe636a9b5ad40
I20240503 14:12:07.649334 28653 ts_tablet_manager.cc:1909] T a8d2600031df4d77b9afe636a9b5ad40 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:07.641594 28604 ts_tablet_manager.cc:1398] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:07.652299 28604 tablet_bootstrap.cc:492] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:07.662256 28661 ts_tablet_manager.cc:1888] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:07.662425 28614 tablet_bootstrap.cc:654] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.664290 28661 log.cc:1196] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/2ef6cd66f34c4980bad5e500309b1568
I20240503 14:12:07.666249 28603 ts_tablet_manager.cc:1888] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:07.668181 28603 log.cc:1196] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/559342d357d64f9eb9e3ab42ccb75942
I20240503 14:12:07.668326 28600 tablet_bootstrap.cc:654] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.668809 28603 ts_tablet_manager.cc:1909] T 559342d357d64f9eb9e3ab42ccb75942 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:07.672695 28661 ts_tablet_manager.cc:1909] T 2ef6cd66f34c4980bad5e500309b1568 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:07.664439 28427 leader_election.cc:341] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:07.674815 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 779f373ef12a4da091309fcf247b5693 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779f373ef12a4da091309fcf247b5693 already in progress: creating tablet
I20240503 14:12:07.673295 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 147) took 4058 ms. Trace:
I20240503 14:12:07.675313 28653 rpcz_store.cc:276] 0503 14:12:03.614488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.614549 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:07.593470 (+3978921us) ts_tablet_manager.cc:1120] Deleting tablet a8d2600031df4d77b9afe636a9b5ad40
0503 14:12:07.593488 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.631847 (+ 38359us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.643520 (+ 11673us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.673207 (+ 29687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3978312}
I20240503 14:12:07.678368 28653 tablet_replica.cc:330] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:07.678663 28653 raft_consensus.cc:2229] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.680516 28600 tablet_bootstrap.cc:492] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:07.680768 28600 ts_tablet_manager.cc:1367] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.003s
I20240503 14:12:07.681922 28600 raft_consensus.cc:348] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.682247 28600 raft_consensus.cc:374] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.682355 28600 raft_consensus.cc:729] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.682372 28026 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_eef00a50c2ce4752ab195301f31eddef [id=acb43ab939ff46428a795f618c6750dd] alter complete (version 3)
I20240503 14:12:07.682705 28600 consensus_queue.cc:260] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.683625 28600 ts_tablet_manager.cc:1398] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:07.684020 28600 tablet_bootstrap.cc:492] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:07.684880 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6665f4b8f8e449a8bbd3d22231c667d8" } new_table_name: "default.Table_253d4e5a90d94897874db169bc3718b6" modify_external_catalogs: true
I20240503 14:12:07.686539 28600 tablet_bootstrap.cc:654] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:07.686594 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f1aa939aced4556ae0f0de0e62d1f97 consensus state has no leader
I20240503 14:12:07.687162 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_507e44ddf53d4f36bf1c7179093063ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.687873 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_507e44ddf53d4f36bf1c7179093063ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:07.689416 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 903f11a608b1483887d9b5234a404470 consensus state has no leader
I20240503 14:12:07.689999 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_26baa53edbc044f4a890a75478267c26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.690693 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26baa53edbc044f4a890a75478267c26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.692679 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 962df44408484e3f8d7b7138f27c1a66 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:07.696357 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a8d2600031df4d77b9afe636a9b5ad40 (table default.Table_22032e3385624e98a1b0bb1484f51c64 [id=304746db399b4284999a5ca79e9e80f4]) successfully deleted
I20240503 14:12:07.699864 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c (DEFAULT_TABLE table=default.Table_d8cb0703edfb4e96ba081bd6f299d867 [id=6119a6b076964647bd7247e0a896a030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.700821 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbcae22ea124edaa14c9409f3052714 (DEFAULT_TABLE table=default.Table_6d7202481c4747ddb44a78ba8bb0063f [id=bafd499c743040d08a2fd8207d4f1e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.700688 28029 catalog_manager.cc:5484] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:07.701297 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 1751f775fe6943d9b5fcc3b113393a73 (table default.Table_a7c8453b862c49c6923695b05befc40b [id=9931742257d74df99c043e569a267ced]). Expected version 1 got 0
I20240503 14:12:07.701714 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920be6d87c4440d0bf7453c86cfe0b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.709677 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fc57203104f749f6be76b0d76e46e8ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.710491 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc57203104f749f6be76b0d76e46e8ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.713487 28181 tablet_service.cc:1467] Processing CreateTablet for tablet e94673f00cb3432185009bda2ce398fa (DEFAULT_TABLE table=default.Table_82fd7fffb051442b8351dccbec8d56ad [id=2849f6ba9ef64d26a5dc65f9ac9048da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.714113 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94673f00cb3432185009bda2ce398fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.716454 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbcae22ea124edaa14c9409f3052714 (DEFAULT_TABLE table=default.Table_6d7202481c4747ddb44a78ba8bb0063f [id=bafd499c743040d08a2fd8207d4f1e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.717089 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbcae22ea124edaa14c9409f3052714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.720629 28465 tablet_service.cc:1467] Processing CreateTablet for tablet e94673f00cb3432185009bda2ce398fa (DEFAULT_TABLE table=default.Table_82fd7fffb051442b8351dccbec8d56ad [id=2849f6ba9ef64d26a5dc65f9ac9048da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.721199 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94673f00cb3432185009bda2ce398fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.723802 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d (DEFAULT_TABLE table=default.Table_32b3856710f446139fa555cd42254eef [id=cf15c50bd96d4420b98a06df477003b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.724362 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe3affd0d144b1fb0078e7690b9b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.724596 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbcae22ea124edaa14c9409f3052714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.727078 29113 raft_consensus.cc:482] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.727350 29113 raft_consensus.cc:504] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.728350 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c (DEFAULT_TABLE table=default.Table_d8cb0703edfb4e96ba081bd6f299d867 [id=6119a6b076964647bd7247e0a896a030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.728699 29113 leader_election.cc:288] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:07.728907 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920be6d87c4440d0bf7453c86cfe0b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.732123 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351d5bfb12c34c5f9de90e2df1b36fc1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:07.732935 28150 leader_election.cc:341] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.734509 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 23186a7a0dfb432e813c8a05e97e46dd (table default.Table_afc737ad630d4de6b239f807771fa6a1 [id=68a4a475afc24b95b5388e43deb4632b]). Expected version 1 got 0
I20240503 14:12:07.734935 28026 catalog_manager.cc:5484] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:07.735666 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8c7d57fc4486457e9848daad79f04356 (table default.Table_67ce7228ef934797ba768f947d2cb0aa [id=5cf73f88a73a4cc5966467ecfff2140f]). Expected version 1 got 0
I20240503 14:12:07.736398 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 151) took 3893 ms. Trace:
I20240503 14:12:07.737874 28603 rpcz_store.cc:276] 0503 14:12:03.794198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.795277 (+ 1079us) service_pool.cc:225] Handling call
0503 14:12:07.365547 (+3570270us) ts_tablet_manager.cc:1120] Deleting tablet 559342d357d64f9eb9e3ab42ccb75942
0503 14:12:07.365558 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.501731 (+136173us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.547924 (+ 46193us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.688034 (+140110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3560728}
I20240503 14:12:07.738317 28603 tablet_replica.cc:330] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:07.738581 28603 raft_consensus.cc:2229] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.738816 28603 raft_consensus.cc:2258] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.739562 28603 ts_tablet_manager.cc:1875] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:07.744944 29157 log.cc:924] Time spent T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240503 14:12:07.747241 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 559342d357d64f9eb9e3ab42ccb75942 (table default.Table_7d8e4d1ecd794871b9d73289f8f43c9a [id=c6805f8e2643425dbed90e5ebe7d3a4a]) successfully deleted
I20240503 14:12:07.748121 28326 tablet_service.cc:1467] Processing CreateTablet for tablet e94673f00cb3432185009bda2ce398fa (DEFAULT_TABLE table=default.Table_82fd7fffb051442b8351dccbec8d56ad [id=2849f6ba9ef64d26a5dc65f9ac9048da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.748625 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94673f00cb3432185009bda2ce398fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:07.752328 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2bba6a3d6e84552b027f23a29cbb38a consensus state has no leader
I20240503 14:12:07.752575 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d (DEFAULT_TABLE table=default.Table_32b3856710f446139fa555cd42254eef [id=cf15c50bd96d4420b98a06df477003b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.752907 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1321728b41b14e7898208cc20c6d1f08" } modify_external_catalogs: true
I20240503 14:12:07.753094 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe3affd0d144b1fb0078e7690b9b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.753194 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1321728b41b14e7898208cc20c6d1f08" } modify_external_catalogs: true
I20240503 14:12:07.755633 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c (DEFAULT_TABLE table=default.Table_d8cb0703edfb4e96ba081bd6f299d867 [id=6119a6b076964647bd7247e0a896a030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.756127 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920be6d87c4440d0bf7453c86cfe0b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.755924 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cc77503fb1ad4d03a6278aef9f3ab979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.756786 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc77503fb1ad4d03a6278aef9f3ab979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.754951 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351d5bfb12c34c5f9de90e2df1b36fc1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:07.764227 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbcae22ea124edaa14c9409f3052714 (DEFAULT_TABLE table=default.Table_6d7202481c4747ddb44a78ba8bb0063f [id=bafd499c743040d08a2fd8207d4f1e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.766084 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbcae22ea124edaa14c9409f3052714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.771936 29142 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:07.777792 28329 tablet_service.cc:1467] Processing CreateTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 (DEFAULT_TABLE table=default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.778196 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 80cd09e769be4bee95bbef9c3c550451 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.778541 28336 tablet_service.cc:1467] Processing CreateTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a (DEFAULT_TABLE table=default.Table_253d4e5a90d94897874db169bc3718b6 [id=ae3e54d70f4541898b7f4d03b4128ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.779130 28333 tablet_service.cc:1467] Processing CreateTablet for tablet d43823da09bb49be8f5d1a1149a4b11d (DEFAULT_TABLE table=default.Table_13c7175b4f704263a322c205dfb42854 [id=99f3436cf65347648070d088f8430e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.780160 28458 tablet_service.cc:1467] Processing CreateTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 (DEFAULT_TABLE table=default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.780630 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1cd28bc07e4ccd9097f6dee471c5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.787082 28466 tablet_service.cc:1467] Processing CreateTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a (DEFAULT_TABLE table=default.Table_253d4e5a90d94897874db169bc3718b6 [id=ae3e54d70f4541898b7f4d03b4128ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.787745 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2bba6a3d6e84552b027f23a29cbb38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.792218 28653 raft_consensus.cc:2258] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.792750 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 80cd09e769be4bee95bbef9c3c550451 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.792886 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1cd28bc07e4ccd9097f6dee471c5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.793325 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cd09e769be4bee95bbef9c3c550451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.794454 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43823da09bb49be8f5d1a1149a4b11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.796221 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2bba6a3d6e84552b027f23a29cbb38a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:07.796187 28150 leader_election.cc:341] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:07.796422 28150 leader_election.cc:302] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:07.796626 29119 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:07.797340 29101 raft_consensus.cc:2726] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:07.797909 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d (DEFAULT_TABLE table=default.Table_32b3856710f446139fa555cd42254eef [id=cf15c50bd96d4420b98a06df477003b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.798713 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fe3affd0d144b1fb0078e7690b9b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.799062 28180 tablet_service.cc:1467] Processing CreateTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 (DEFAULT_TABLE table=default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.799535 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1cd28bc07e4ccd9097f6dee471c5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.799903 28456 tablet_service.cc:1467] Processing CreateTablet for tablet d43823da09bb49be8f5d1a1149a4b11d (DEFAULT_TABLE table=default.Table_13c7175b4f704263a322c205dfb42854 [id=99f3436cf65347648070d088f8430e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.800418 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43823da09bb49be8f5d1a1149a4b11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.802078 28188 tablet_service.cc:1467] Processing CreateTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a (DEFAULT_TABLE table=default.Table_253d4e5a90d94897874db169bc3718b6 [id=ae3e54d70f4541898b7f4d03b4128ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.802596 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2bba6a3d6e84552b027f23a29cbb38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.804589 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 80cd09e769be4bee95bbef9c3c550451 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.805179 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cd09e769be4bee95bbef9c3c550451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.808599 28182 tablet_service.cc:1467] Processing CreateTablet for tablet d43823da09bb49be8f5d1a1149a4b11d (DEFAULT_TABLE table=default.Table_13c7175b4f704263a322c205dfb42854 [id=99f3436cf65347648070d088f8430e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.809163 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43823da09bb49be8f5d1a1149a4b11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.813411 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cd09e769be4bee95bbef9c3c550451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.796185 28653 ts_tablet_manager.cc:1875] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.831008 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 140de4d395d54b7696062ed96b8aa89c
I20240503 14:12:07.832041 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_aa957a5e7a874114aaaec61023360b59" } modify_external_catalogs: true
I20240503 14:12:07.832486 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_aa957a5e7a874114aaaec61023360b59" } modify_external_catalogs: true
I20240503 14:12:07.834616 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_10446f75b72747c0bfa9aa56408f9cd0" } modify_external_catalogs: true
I20240503 14:12:07.834854 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_10446f75b72747c0bfa9aa56408f9cd0" } modify_external_catalogs: true
I20240503 14:12:07.836920 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7c4e797497e74bbe98ff720b9ffd05ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.837728 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c4e797497e74bbe98ff720b9ffd05ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.840879 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 140de4d395d54b7696062ed96b8aa89c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.842046 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet 140de4d395d54b7696062ed96b8aa89c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.842550 29113 raft_consensus.cc:482] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.842811 29113 raft_consensus.cc:504] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:07.843032 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fb279b38e7764d3bac7f2aa1667ca58e" } modify_external_catalogs: true
I20240503 14:12:07.843326 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fb279b38e7764d3bac7f2aa1667ca58e" } modify_external_catalogs: true
I20240503 14:12:07.843724 29113 leader_election.cc:288] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:07.845301 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 962df44408484e3f8d7b7138f27c1a66
I20240503 14:12:07.845868 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 962df44408484e3f8d7b7138f27c1a66 deleted and replaced by tablet e4a544fc27624dcc8a4cb003e59408f2
I20240503 14:12:07.846331 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_965ce30debbf4cc6a42b1fee93fb54de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.847052 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_965ce30debbf4cc6a42b1fee93fb54de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.851300 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 962df44408484e3f8d7b7138f27c1a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e4a544fc27624dcc8a4cb003e59408f2) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:07.852046 28455 tablet_service.cc:1514] Processing DeleteTablet for tablet 962df44408484e3f8d7b7138f27c1a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e4a544fc27624dcc8a4cb003e59408f2) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.852286 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_82fd7fffb051442b8351dccbec8d56ad" } new_table_name: "default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d" modify_external_catalogs: true
I20240503 14:12:07.853472 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet 962df44408484e3f8d7b7138f27c1a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e4a544fc27624dcc8a4cb003e59408f2) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:07.868634 28025 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 993bee22c09c424399d54756701c5517
I20240503 14:12:07.874480 28603 ts_tablet_manager.cc:1888] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:07.662631 28604 tablet_bootstrap.cc:654] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.880883 29107 raft_consensus.cc:482] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.884613 29107 raft_consensus.cc:504] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.885545 29107 leader_election.cc:288] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:07.886883 28603 log.cc:1196] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ab05178dc081480ba409aab7f143f849
I20240503 14:12:07.887046 28653 ts_tablet_manager.cc:1888] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:07.887284 28653 log.cc:1196] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a2bfdcb8e2b241ca9a6922fd8491e27b
I20240503 14:12:07.887499 28603 ts_tablet_manager.cc:1909] T ab05178dc081480ba409aab7f143f849 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:07.887876 28653 ts_tablet_manager.cc:1909] T a2bfdcb8e2b241ca9a6922fd8491e27b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:07.888214 28290 leader_election.cc:341] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a0e8b0eed57c4ad1bf6040507f6f8d4c
I20240503 14:12:07.890283 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 146) took 4276 ms. Trace:
I20240503 14:12:07.891570 28653 rpcz_store.cc:276] 0503 14:12:03.614326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.615960 (+ 1634us) service_pool.cc:225] Handling call
0503 14:12:07.678180 (+4062220us) ts_tablet_manager.cc:1120] Deleting tablet a2bfdcb8e2b241ca9a6922fd8491e27b
0503 14:12:07.678196 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.871683 (+193487us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.876010 (+ 4327us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.890230 (+ 14220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4045992}
I20240503 14:12:07.893107 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 149) took 4092 ms. Trace:
I20240503 14:12:07.893805 28653 rpcz_store.cc:276] 0503 14:12:03.801132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.801206 (+ 74us) service_pool.cc:225] Handling call
0503 14:12:07.892863 (+4091657us) ts_tablet_manager.cc:1120] Deleting tablet 93462d2b1a474bd688d6266cd3b0695c
0503 14:12:07.892876 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.893041 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4065171}
I20240503 14:12:07.894821 28653 tablet_replica.cc:330] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:07.895407 28653 raft_consensus.cc:2229] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.898929 28653 pending_rounds.cc:62] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:07.900548 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8b0eed57c4ad1bf6040507f6f8d4c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:07.901273 28489 raft_consensus.cc:2454] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:07.902123 28290 leader_election.cc:302] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:07.902696 29107 raft_consensus.cc:2781] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.902870 29107 raft_consensus.cc:482] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.903017 29107 raft_consensus.cc:3037] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.904320 28653 pending_rounds.cc:68] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597671148142592 op_type: NO_OP noop_request { }
I20240503 14:12:07.904601 28653 raft_consensus.cc:2866] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:07.905277 28653 raft_consensus.cc:2258] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.907191 28653 ts_tablet_manager.cc:1875] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.908430 29107 raft_consensus.cc:504] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.909508 29107 leader_election.cc:288] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:07.911065 28290 leader_election.cc:341] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a0e8b0eed57c4ad1bf6040507f6f8d4c
I20240503 14:12:07.912616 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8b0eed57c4ad1bf6040507f6f8d4c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:07.913105 28489 raft_consensus.cc:3037] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.913049 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 4fad4508a29d4822b81af6947b57e7ef (DEFAULT_TABLE table=default.Table_68a22cfee7f644fcba9b9c3e34e6205c [id=0a3978674b74406484d7b5e9880f6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.917327 28489 raft_consensus.cc:2454] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:07.919922 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a2bfdcb8e2b241ca9a6922fd8491e27b (table default.Table_3a2892577d0744b5927abb18464fb658 [id=bd615ce41c41475fa37c3d6de18ece3a]) successfully deleted
W20240503 14:12:07.920284 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 93462d2b1a474bd688d6266cd3b0695c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93462d2b1a474bd688d6266cd3b0695c already in progress: creating tablet
I20240503 14:12:07.920874 28290 leader_election.cc:302] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:07.921499 29107 raft_consensus.cc:2781] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.921474 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 4fad4508a29d4822b81af6947b57e7ef (DEFAULT_TABLE table=default.Table_68a22cfee7f644fcba9b9c3e34e6205c [id=0a3978674b74406484d7b5e9880f6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.921702 29107 raft_consensus.cc:686] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:07.922231 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fad4508a29d4822b81af6947b57e7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.922129 29107 consensus_queue.cc:237] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.922653 29153 consensus_queue.cc:1035] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:07.922883 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 140de4d395d54b7696062ed96b8aa89c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:07.928570 28604 tablet_bootstrap.cc:492] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:07.928921 28604 ts_tablet_manager.cc:1367] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.277s user 0.004s sys 0.003s
I20240503 14:12:07.930774 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 167) took 3479 ms. Trace:
I20240503 14:12:07.933207 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 4fad4508a29d4822b81af6947b57e7ef (DEFAULT_TABLE table=default.Table_68a22cfee7f644fcba9b9c3e34e6205c [id=0a3978674b74406484d7b5e9880f6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.939621 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fad4508a29d4822b81af6947b57e7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.939914 28661 rpcz_store.cc:276] 0503 14:12:04.451182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.770987 (+319805us) service_pool.cc:225] Handling call
0503 14:12:07.619089 (+2848102us) ts_tablet_manager.cc:1120] Deleting tablet 2ef6cd66f34c4980bad5e500309b1568
0503 14:12:07.619102 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.646213 (+ 27111us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.662087 (+ 15874us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.930673 (+268586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2536608}
W20240503 14:12:07.940595 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet f45eb624434a404d85dcd233721ebf68 (table default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f45eb624434a404d85dcd233721ebf68 consensus state has no leader
I20240503 14:12:07.945194 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 170) took 2832 ms. Trace:
I20240503 14:12:07.953213 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:07.955030 28653 ts_tablet_manager.cc:1888] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:07.955246 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.958163 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 2ef6cd66f34c4980bad5e500309b1568 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]) successfully deleted
I20240503 14:12:07.958323 28653 log.cc:1196] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7de7832122804eada175f710cc5bab5e
I20240503 14:12:07.958465 28661 rpcz_store.cc:276] 0503 14:12:05.113138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.113207 (+ 69us) service_pool.cc:225] Handling call
0503 14:12:07.944931 (+2831724us) ts_tablet_manager.cc:1120] Deleting tablet a6dbb59d98834ab6bbac4abfa92681e3
0503 14:12:07.944945 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.945094 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2831339}
I20240503 14:12:07.959349 28653 ts_tablet_manager.cc:1909] T 7de7832122804eada175f710cc5bab5e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:07.961300 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 148) took 4164 ms. Trace:
I20240503 14:12:07.961488 28653 rpcz_store.cc:276] 0503 14:12:03.796413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.798256 (+ 1843us) service_pool.cc:225] Handling call
0503 14:12:07.894719 (+4096463us) ts_tablet_manager.cc:1120] Deleting tablet 7de7832122804eada175f710cc5bab5e
0503 14:12:07.894730 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.935598 (+ 40868us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.954840 (+ 19242us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.961236 (+ 6396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4066816}
I20240503 14:12:07.961977 28653 tablet_replica.cc:330] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:07.962282 28653 raft_consensus.cc:2229] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.962702 28653 raft_consensus.cc:2258] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.959211 28604 raft_consensus.cc:348] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.962309 29153 raft_consensus.cc:482] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.976815 28604 raft_consensus.cc:374] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.976822 29153 raft_consensus.cc:504] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.977059 28604 raft_consensus.cc:729] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.960183 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_327bf3afee7847bd9497b8340b940508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:07.978416 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_327bf3afee7847bd9497b8340b940508 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:07.962980 28031 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d6f593467bf4482cb926b63480790ad4
I20240503 14:12:07.980280 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_26baa53edbc044f4a890a75478267c26" } new_table_name: "default.Table_a9f63e6f47cb4390b4fff24fd5d4f83e" modify_external_catalogs: true
I20240503 14:12:07.980571 28653 ts_tablet_manager.cc:1875] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:07.981634 29153 leader_election.cc:288] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:07.981662 28604 consensus_queue.cc:260] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.982064 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde0e59d911d4c1eb5125b1f869b396c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:07.982399 28217 raft_consensus.cc:2454] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:07.963263 28030 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fa856c156aa14327a1ce0962b6ab6e34
I20240503 14:12:07.983546 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_419d8eaa44c34cca86e012471fd9814d" } new_table_name: "default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8" modify_external_catalogs: true
W20240503 14:12:07.987715 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 190507us
I20240503 14:12:07.989027 28614 tablet_bootstrap.cc:492] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:07.989292 28614 ts_tablet_manager.cc:1367] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.383s user 0.012s sys 0.008s
I20240503 14:12:07.990381 28614 raft_consensus.cc:348] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.990487 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7de7832122804eada175f710cc5bab5e (table default.Table_cc4fe0dac143484396ef862744cc9967 [id=e324426e1ff447a38c25b3867724fc4c]) successfully deleted
I20240503 14:12:07.990649 28614 raft_consensus.cc:374] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.990887 28614 raft_consensus.cc:729] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
W20240503 14:12:07.990998 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a6dbb59d98834ab6bbac4abfa92681e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6dbb59d98834ab6bbac4abfa92681e3 already in progress: creating tablet
I20240503 14:12:07.991312 28614 consensus_queue.cc:260] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:07.992120 28614 ts_tablet_manager.cc:1398] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:07.995877 28653 ts_tablet_manager.cc:1888] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:07.996575 28604 ts_tablet_manager.cc:1398] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20240503 14:12:07.997138 28604 tablet_bootstrap.cc:492] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:07.997341 28427 leader_election.cc:302] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:08.001143 29153 raft_consensus.cc:2781] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.001257 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51834695b8b4a729a088da81eee2e47" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:08.001406 29153 raft_consensus.cc:482] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.001564 29153 raft_consensus.cc:3037] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.003604 29162 raft_consensus.cc:482] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.008016 29162 raft_consensus.cc:504] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.009063 29162 leader_election.cc:288] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:08.007752 28150 leader_election.cc:341] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.010190 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a91f22268346b38ae35f96b0d763f7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:08.011680 28150 leader_election.cc:341] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.011606 29153 raft_consensus.cc:504] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.011972 28653 log.cc:1196] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2ef6cd66f34c4980bad5e500309b1568
I20240503 14:12:08.012692 28653 ts_tablet_manager.cc:1909] T 2ef6cd66f34c4980bad5e500309b1568 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.013628 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 151) took 4111 ms. Trace:
I20240503 14:12:08.013796 28653 rpcz_store.cc:276] 0503 14:12:03.902701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.902789 (+ 88us) service_pool.cc:225] Handling call
0503 14:12:07.961850 (+4059061us) ts_tablet_manager.cc:1120] Deleting tablet 2ef6cd66f34c4980bad5e500309b1568
0503 14:12:07.961865 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.992109 (+ 30244us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.995736 (+ 3627us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.013565 (+ 17829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3636808}
I20240503 14:12:08.014369 28653 tablet_replica.cc:330] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.014648 28653 raft_consensus.cc:2229] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.014957 28653 raft_consensus.cc:2258] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.015681 28653 ts_tablet_manager.cc:1875] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.015933 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde0e59d911d4c1eb5125b1f869b396c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:08.016281 28217 raft_consensus.cc:3037] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.019726 28217 raft_consensus.cc:2454] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:08.020537 28427 leader_election.cc:302] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:08.020979 29030 raft_consensus.cc:2781] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.021131 29030 raft_consensus.cc:686] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:08.021467 29030 consensus_queue.cc:237] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.024145 28653 ts_tablet_manager.cc:1888] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:08.025317 28653 log.cc:1196] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/b166fe6366ba496a869af1b106fe9d8e
I20240503 14:12:08.025934 28604 tablet_bootstrap.cc:654] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.026105 28653 ts_tablet_manager.cc:1909] T b166fe6366ba496a869af1b106fe9d8e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.027010 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 150) took 4128 ms. Trace:
I20240503 14:12:08.027242 28653 rpcz_store.cc:276] 0503 14:12:03.898945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.899300 (+ 355us) service_pool.cc:225] Handling call
0503 14:12:08.014233 (+4114933us) ts_tablet_manager.cc:1120] Deleting tablet b166fe6366ba496a869af1b106fe9d8e
0503 14:12:08.014243 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.019846 (+ 5603us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.023937 (+ 4091us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.026957 (+ 3020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3686066}
I20240503 14:12:08.027549 29153 leader_election.cc:288] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:08.028072 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 171) took 3661 ms. Trace:
I20240503 14:12:08.028303 28653 rpcz_store.cc:276] 0503 14:12:04.366978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.367217 (+ 239us) service_pool.cc:225] Handling call
0503 14:12:08.027818 (+3660601us) ts_tablet_manager.cc:1120] Deleting tablet 2ef6cd66f34c4980bad5e500309b1568
0503 14:12:08.027830 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.027939 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3660347}
I20240503 14:12:08.029026 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 172) took 3661 ms. Trace:
I20240503 14:12:08.029230 28653 rpcz_store.cc:276] 0503 14:12:04.367156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.367517 (+ 361us) service_pool.cc:225] Handling call
0503 14:12:08.028830 (+3661313us) ts_tablet_manager.cc:1120] Deleting tablet 265dc5f2630e453ea71abddb1a9753f3
0503 14:12:08.028841 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.028958 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3661130}
I20240503 14:12:08.029770 28653 tablet_replica.cc:330] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.030110 28653 raft_consensus.cc:2229] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.030385 28653 raft_consensus.cc:2258] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.031391 28653 ts_tablet_manager.cc:1875] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.033041 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 168) took 3581 ms. Trace:
I20240503 14:12:08.034013 28661 rpcz_store.cc:276] 0503 14:12:04.451832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.771022 (+319190us) service_pool.cc:225] Handling call
0503 14:12:07.977775 (+3206753us) ts_tablet_manager.cc:1120] Deleting tablet 265dc5f2630e453ea71abddb1a9753f3
0503 14:12:07.977788 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.032922 (+ 55134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2862960}
W20240503 14:12:08.036643 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 265dc5f2630e453ea71abddb1a9753f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 265dc5f2630e453ea71abddb1a9753f3 already in progress: creating tablet
I20240503 14:12:08.037241 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fad4508a29d4822b81af6947b57e7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.037016 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2ef6cd66f34c4980bad5e500309b1568 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]) successfully deleted
I20240503 14:12:08.037597 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_17a1019962da48c0992b5452d4a67aae [id=548632b6d08e4435ad561b074e5e5b14]) successfully deleted
W20240503 14:12:08.037868 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2ef6cd66f34c4980bad5e500309b1568 because the tablet was not found. No further retry: Not found: Tablet not found: 2ef6cd66f34c4980bad5e500309b1568
W20240503 14:12:08.038125 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 265dc5f2630e453ea71abddb1a9753f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 265dc5f2630e453ea71abddb1a9753f3 already in progress: creating tablet
I20240503 14:12:08.036726 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 171) took 2910 ms. Trace:
I20240503 14:12:08.038441 28661 rpcz_store.cc:276] 0503 14:12:05.125841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.128776 (+ 2935us) service_pool.cc:225] Handling call
0503 14:12:08.036440 (+2907664us) ts_tablet_manager.cc:1120] Deleting tablet ed9262a7e828489ab51a745068c737df
0503 14:12:08.036453 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.036625 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2907258}
W20240503 14:12:08.040424 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ed9262a7e828489ab51a745068c737df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9262a7e828489ab51a745068c737df already in progress: creating tablet
I20240503 14:12:08.041683 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3703ab7db61b4c9ead3058bfdc123189
I20240503 14:12:08.042608 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:08.043242 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde0e59d911d4c1eb5125b1f869b396c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:08.043632 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde0e59d911d4c1eb5125b1f869b396c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.044831 28614 tablet_bootstrap.cc:492] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:08.045658 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.046990 28614 tablet_bootstrap.cc:654] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.049206 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.049185 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51834695b8b4a729a088da81eee2e47" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.050122 29135 tablet.cc:1722] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:08.050381 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a91f22268346b38ae35f96b0d763f7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.052568 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:08.055269 28427 leader_election.cc:341] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:08.055717 28427 leader_election.cc:341] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.061466 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0] alter complete (version 2)
W20240503 14:12:08.062062 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e94673f00cb3432185009bda2ce398fa consensus state has no leader
I20240503 14:12:08.056270 29153 consensus_queue.cc:1035] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.075877 28614 tablet_bootstrap.cc:492] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:08.076234 28614 ts_tablet_manager.cc:1367] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.002s
I20240503 14:12:08.081730 28614 raft_consensus.cc:348] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.083624 28614 raft_consensus.cc:374] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.084209 28614 raft_consensus.cc:729] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.085891 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 152) took 4291 ms. Trace:
I20240503 14:12:08.088543 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9d8f6693297844bc907fc355a5561203" } new_table_name: "default.Table_6673b8d0a18d42c3a18f6e611c596580" modify_external_catalogs: true
W20240503 14:12:08.093735 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77c77a11ef384709aef5b62273380573 has no consensus state
I20240503 14:12:08.093909 28603 rpcz_store.cc:276] 0503 14:12:03.794321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:03.795313 (+ 992us) service_pool.cc:225] Handling call
0503 14:12:07.738215 (+3942902us) ts_tablet_manager.cc:1120] Deleting tablet ab05178dc081480ba409aab7f143f849
0503 14:12:07.738227 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:07.811751 (+ 73524us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:07.874316 (+ 62565us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.085778 (+211462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3932393}
I20240503 14:12:08.094177 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet 2f89fca0480d4bf28160384ae0421bce from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:08.094516 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ab05178dc081480ba409aab7f143f849 (table default.Table_adb8e4f58ba042f0a57523a25228ca92 [id=37c31c087c65469f96f4322d84ba161f]) successfully deleted
I20240503 14:12:08.085034 28614 consensus_queue.cc:260] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.095855 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet bd76060daab84c2f9f6727b8a9bb3a49 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:08.096056 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.096414 28614 ts_tablet_manager.cc:1398] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240503 14:12:08.096800 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a91f22268346b38ae35f96b0d763f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.097525 28614 tablet_bootstrap.cc:492] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:08.096843 28603 tablet_replica.cc:330] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.099195 28603 raft_consensus.cc:2229] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.099610 28603 raft_consensus.cc:2258] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:08.100124 28150 leader_election.cc:341] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.100369 28150 leader_election.cc:302] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:08.100728 29162 raft_consensus.cc:2726] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:08.101089 28150 leader_election.cc:341] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.101284 28150 leader_election.cc:302] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:08.101600 29162 raft_consensus.cc:2726] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:08.101840 28603 ts_tablet_manager.cc:1875] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.103719 28614 tablet_bootstrap.cc:654] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.104991 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_77748b64009a403a8a014fa47bff52b4" } new_table_name: "default.Table_80d78826082349c882ea2c0ca66e0141" modify_external_catalogs: true
I20240503 14:12:08.105219 28653 ts_tablet_manager.cc:1888] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.105406 28653 log.cc:1196] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/779f373ef12a4da091309fcf247b5693
I20240503 14:12:08.105880 28653 ts_tablet_manager.cc:1909] T 779f373ef12a4da091309fcf247b5693 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.106848 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 173) took 3727 ms. Trace:
I20240503 14:12:08.107033 28653 rpcz_store.cc:276] 0503 14:12:04.379628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.379696 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:08.029672 (+3649976us) ts_tablet_manager.cc:1120] Deleting tablet 779f373ef12a4da091309fcf247b5693
0503 14:12:08.029682 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.098700 (+ 69018us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.105073 (+ 6373us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.106784 (+ 1711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3649704}
I20240503 14:12:08.107312 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_aa516ed89b714cf788cdf47928d6a415"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:08.107717 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 174) took 3018 ms. Trace:
I20240503 14:12:08.107880 28653 rpcz_store.cc:276] 0503 14:12:05.088879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.090194 (+ 1315us) service_pool.cc:225] Handling call
0503 14:12:08.107419 (+3017225us) ts_tablet_manager.cc:1120] Deleting tablet a6dbb59d98834ab6bbac4abfa92681e3
0503 14:12:08.107430 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.107624 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3009696}
I20240503 14:12:08.108521 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 779f373ef12a4da091309fcf247b5693 (table default.Table_8c90a0cf9e1f495ea0ea85f1f8726f52 [id=857b2f49969d4291b98de4f3776e627d]) successfully deleted
W20240503 14:12:08.108870 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa516ed89b714cf788cdf47928d6a415 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.112923 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 175) took 2965 ms. Trace:
W20240503 14:12:08.114248 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a6dbb59d98834ab6bbac4abfa92681e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6dbb59d98834ab6bbac4abfa92681e3 already in progress: creating tablet
I20240503 14:12:08.114552 28653 rpcz_store.cc:276] 0503 14:12:05.147988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.148163 (+ 175us) service_pool.cc:225] Handling call
0503 14:12:08.112658 (+2964495us) ts_tablet_manager.cc:1120] Deleting tablet ed9262a7e828489ab51a745068c737df
0503 14:12:08.112669 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.112829 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2962637}
W20240503 14:12:08.114754 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 651478ef632144079f93f3b50bff4611 consensus state has no leader
I20240503 14:12:08.115401 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_adc1bb9599f646e2bf55dcf13fdf6688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:08.116142 28653 tablet_replica.cc:330] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.116662 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet dc337647c7cd402e91720478380f4696 (table default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd]). Expected version 4 got 3
I20240503 14:12:08.119238 28653 raft_consensus.cc:2229] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:08.119846 28653 raft_consensus.cc:2258] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.125844 28600 tablet_bootstrap.cc:492] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.126690 28600 ts_tablet_manager.cc:1367] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.443s user 0.003s sys 0.021s
I20240503 14:12:08.127920 28600 raft_consensus.cc:348] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.128288 28653 ts_tablet_manager.cc:1875] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.128432 28600 raft_consensus.cc:374] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.128641 28600 raft_consensus.cc:729] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.129145 28600 consensus_queue.cc:260] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:08.129344 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet daf5d153ab5142c0902cc6e96629f412 consensus state has no leader
I20240503 14:12:08.129936 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet bf21f6234dd64cf8a32f1599fe50746e from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:08.130313 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adc1bb9599f646e2bf55dcf13fdf6688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:08.134426 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ed9262a7e828489ab51a745068c737df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9262a7e828489ab51a745068c737df already in progress: creating tablet
I20240503 14:12:08.135902 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8b0eed57c4ad1bf6040507f6f8d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:01 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.137745 28600 ts_tablet_manager.cc:1398] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:12:08.142743 28603 ts_tablet_manager.cc:1888] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:08.143019 27851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu pid 27927
I20240503 14:12:08.144485 27991 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:08.150628 28653 ts_tablet_manager.cc:1888] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.151886 28600 tablet_bootstrap.cc:492] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.153738 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c7c5dfac02d4dfe932996d2a37fe031 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:08.154343 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b1728b6f0e64661aaa8184ce51366d2 consensus state has no leader
I20240503 14:12:08.155716 28600 tablet_bootstrap.cc:654] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.152112 28653 log.cc:1196] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f45eb624434a404d85dcd233721ebf68
I20240503 14:12:08.156651 28653 ts_tablet_manager.cc:1909] T f45eb624434a404d85dcd233721ebf68 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.157675 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 176) took 2898 ms. Trace:
I20240503 14:12:08.157841 28653 rpcz_store.cc:276] 0503 14:12:05.259029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.259095 (+ 66us) service_pool.cc:225] Handling call
0503 14:12:08.115769 (+2856674us) ts_tablet_manager.cc:1120] Deleting tablet f45eb624434a404d85dcd233721ebf68
0503 14:12:08.115784 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.141015 (+ 25231us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.150430 (+ 9415us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.157616 (+ 7186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2856402}
I20240503 14:12:08.158674 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 177) took 2898 ms. Trace:
I20240503 14:12:08.158845 28653 rpcz_store.cc:276] 0503 14:12:05.260273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.260332 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:08.158390 (+2898058us) ts_tablet_manager.cc:1120] Deleting tablet bd76060daab84c2f9f6727b8a9bb3a49
0503 14:12:08.158403 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.158555 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2897733}
I20240503 14:12:08.158727 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7fdf02629122402c9cbf8053f89eced3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:08.159534 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 178) took 2897 ms. Trace:
I20240503 14:12:08.159703 28653 rpcz_store.cc:276] 0503 14:12:05.262230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.262289 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:08.159315 (+2897026us) ts_tablet_manager.cc:1120] Deleting tablet 63e977f7837e49fb93ef0fd18b619792
0503 14:12:08.159326 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.159479 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896754}
I20240503 14:12:08.160328 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 179) took 2753 ms. Trace:
I20240503 14:12:08.160490 28653 rpcz_store.cc:276] 0503 14:12:05.407212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.407443 (+ 231us) service_pool.cc:225] Handling call
0503 14:12:08.160142 (+2752699us) ts_tablet_manager.cc:1120] Deleting tablet 7c6f41bb9c5d40b0913264864ecc3528
0503 14:12:08.160153 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.160270 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2752276}
I20240503 14:12:08.162566 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7c8453b862c49c6923695b05befc40b" } modify_external_catalogs: true
W20240503 14:12:08.162832 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 63e977f7837e49fb93ef0fd18b619792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e977f7837e49fb93ef0fd18b619792 already in progress: creating tablet
I20240503 14:12:08.162881 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7c8453b862c49c6923695b05befc40b" } modify_external_catalogs: true
W20240503 14:12:08.163672 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7c6f41bb9c5d40b0913264864ecc3528 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6f41bb9c5d40b0913264864ecc3528 already in progress: creating tablet
I20240503 14:12:08.164402 28025 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_67ce7228ef934797ba768f947d2cb0aa [id=5cf73f88a73a4cc5966467ecfff2140f] alter complete (version 1)
I20240503 14:12:08.167300 28604 tablet_bootstrap.cc:492] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:08.166925 29107 raft_consensus.cc:482] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a64bc15d65d84ad5a005960a9a84df9c)
I20240503 14:12:08.168287 29107 raft_consensus.cc:504] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.169198 29107 leader_election.cc:288] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:08.170271 28290 leader_election.cc:341] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: b1de9717d0e44179b875174d5a76e4c3
I20240503 14:12:08.170660 28603 log.cc:1196] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/b166fe6366ba496a869af1b106fe9d8e
W20240503 14:12:08.170655 28290 leader_election.cc:341] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b1de9717d0e44179b875174d5a76e4c3
I20240503 14:12:08.171962 28290 leader_election.cc:302] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:08.172587 29107 raft_consensus.cc:2726] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:12:08.172856 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fdf02629122402c9cbf8053f89eced3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.175746 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ede6e833710649e08e31516f0890c2d6" } modify_external_catalogs: true
I20240503 14:12:08.176138 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ede6e833710649e08e31516f0890c2d6" } modify_external_catalogs: true
I20240503 14:12:08.176937 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 903f11a608b1483887d9b5234a404470 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:08.177037 28603 ts_tablet_manager.cc:1909] T b166fe6366ba496a869af1b106fe9d8e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.177480 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 1751f775fe6943d9b5fcc3b113393a73 (table default.Table_a7c8453b862c49c6923695b05befc40b [id=9931742257d74df99c043e569a267ced]). Expected version 1 got 0
I20240503 14:12:08.179569 28653 tablet_replica.cc:330] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.179754 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 153) took 3725 ms. Trace:
I20240503 14:12:08.179920 28653 raft_consensus.cc:2229] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:08.179827 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_027b06bfe8c444dcba6cd5d6f42220ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:08.180490 28025 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd] alter complete (version 4)
W20240503 14:12:08.180590 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_027b06bfe8c444dcba6cd5d6f42220ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.181077 28603 rpcz_store.cc:276] 0503 14:12:04.454765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.455731 (+ 966us) service_pool.cc:225] Handling call
0503 14:12:08.096705 (+3640974us) ts_tablet_manager.cc:1120] Deleting tablet b166fe6366ba496a869af1b106fe9d8e
0503 14:12:08.096718 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.129085 (+ 32367us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.142540 (+ 13455us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.179703 (+ 37163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":621,"tablet-delete.queue_time_us":3640663}
I20240503 14:12:08.178376 28614 tablet_bootstrap.cc:492] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:08.181747 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet b166fe6366ba496a869af1b106fe9d8e (table default.Table_17a1019962da48c0992b5452d4a67aae [id=548632b6d08e4435ad561b074e5e5b14]) successfully deleted
I20240503 14:12:08.181818 28614 ts_tablet_manager.cc:1367] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.002s
I20240503 14:12:08.184402 28653 raft_consensus.cc:2258] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.185236 28653 ts_tablet_manager.cc:1875] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.186028 28614 raft_consensus.cc:348] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.186318 28614 raft_consensus.cc:374] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.186462 28614 raft_consensus.cc:729] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.186743 28614 consensus_queue.cc:260] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.188007 29030 raft_consensus.cc:482] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.188325 29030 raft_consensus.cc:504] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.189996 28603 tablet_replica.cc:330] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.191295 28603 raft_consensus.cc:2229] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.191856 28603 raft_consensus.cc:2258] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.192320 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e355aafb2f5b4337bedaac6fe5539dc2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.193794 28614 ts_tablet_manager.cc:1398] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240503 14:12:08.194819 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e355aafb2f5b4337bedaac6fe5539dc2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:08.193274 28603 ts_tablet_manager.cc:1875] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.195129 28217 raft_consensus.cc:2454] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:08.196190 28427 leader_election.cc:302] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:08.196600 29153 raft_consensus.cc:2781] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:12:08.196667 28427 leader_election.cc:341] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.196777 29153 raft_consensus.cc:482] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.196895 29153 raft_consensus.cc:3037] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.198477 29030 leader_election.cc:288] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:08.199033 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b86f741291e64aef9908c5e036e4cce8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.199829 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b86f741291e64aef9908c5e036e4cce8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.200786 29153 raft_consensus.cc:504] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.201953 29153 leader_election.cc:288] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:08.202522 28614 tablet_bootstrap.cc:492] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:08.204205 28600 tablet_bootstrap.cc:492] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.204504 28600 ts_tablet_manager.cc:1367] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.005s
I20240503 14:12:08.205686 28600 raft_consensus.cc:348] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.206075 28600 raft_consensus.cc:374] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.206198 28600 raft_consensus.cc:729] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.206426 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e355aafb2f5b4337bedaac6fe5539dc2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:08.206586 28600 consensus_queue.cc:260] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.206734 28217 raft_consensus.cc:3037] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.207209 28600 ts_tablet_manager.cc:1398] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:08.208266 28600 tablet_bootstrap.cc:492] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.208645 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e355aafb2f5b4337bedaac6fe5539dc2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:08.209667 28427 leader_election.cc:341] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.211815 28614 tablet_bootstrap.cc:654] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.215967 28600 tablet_bootstrap.cc:654] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.218410 28217 raft_consensus.cc:2454] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:08.219250 28427 leader_election.cc:302] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:08.219801 29153 raft_consensus.cc:2781] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.220039 29153 raft_consensus.cc:686] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:08.167529 28661 tablet_replica.cc:330] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:08.220548 29153 consensus_queue.cc:237] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.222275 28603 ts_tablet_manager.cc:1888] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.223347 28661 raft_consensus.cc:2229] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.224365 28661 raft_consensus.cc:2258] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.231042 28180 tablet_service.cc:1467] Processing CreateTablet for tablet ff1958dc488e4854a8c8e74da9fab243 (DEFAULT_TABLE table=default.Table_fc57203104f749f6be76b0d76e46e8ee [id=aed23d17e9704c36b4b4ecb604987e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.231662 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1958dc488e4854a8c8e74da9fab243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.233404 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 77c77a11ef384709aef5b62273380573 (DEFAULT_TABLE table=default.Table_a9f63e6f47cb4390b4fff24fd5d4f83e [id=2edd1164c07a4a67b51262d402d53596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.233951 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c77a11ef384709aef5b62273380573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.236500 28182 tablet_service.cc:1467] Processing CreateTablet for tablet d93fea5caec94695a5df26d3630c50c7 (DEFAULT_TABLE table=default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.237007 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93fea5caec94695a5df26d3630c50c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.239459 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a (DEFAULT_TABLE table=default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.239935 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b176a071ba4f3fbff2a64d6e46027a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.240536 28460 tablet_service.cc:1467] Processing CreateTablet for tablet ff1958dc488e4854a8c8e74da9fab243 (DEFAULT_TABLE table=default.Table_fc57203104f749f6be76b0d76e46e8ee [id=aed23d17e9704c36b4b4ecb604987e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.241241 28191 tablet_service.cc:1467] Processing CreateTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 (DEFAULT_TABLE table=default.Table_507e44ddf53d4f36bf1c7179093063ff [id=6539aa29cb864a2fa19c8a2f1ac7bd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.241750 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd405fd84d44076b1792b7fcbe00e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.244534 28188 tablet_service.cc:1467] Processing CreateTablet for tablet e4a544fc27624dcc8a4cb003e59408f2 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.244997 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a544fc27624dcc8a4cb003e59408f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.245692 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.231369 28335 tablet_service.cc:1467] Processing CreateTablet for tablet ff1958dc488e4854a8c8e74da9fab243 (DEFAULT_TABLE table=default.Table_fc57203104f749f6be76b0d76e46e8ee [id=aed23d17e9704c36b4b4ecb604987e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.246165 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8411f474c247c2abfd0bd9682fb8a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.246245 28603 log.cc:1196] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/2ef6cd66f34c4980bad5e500309b1568
I20240503 14:12:08.246480 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1958dc488e4854a8c8e74da9fab243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.247658 28603 ts_tablet_manager.cc:1909] T 2ef6cd66f34c4980bad5e500309b1568 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.248517 28192 tablet_service.cc:1467] Processing CreateTablet for tablet a05f786073084bd1b8c472c20dc8bff6 (DEFAULT_TABLE table=default.Table_f010ec576f0a40cb91c71b23da613670 [id=3221ffb751294f5b9f7fa7848d9845e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.248978 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05f786073084bd1b8c472c20dc8bff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.249671 28179 tablet_service.cc:1467] Processing CreateTablet for tablet b75036cdf9e04ea4aa767fd12016f177 (DEFAULT_TABLE table=default.Table_55ff9f3cd6f84368add48d9cecc3fb1b [id=30ff854aabe84b2cb03824d287ae5685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.250186 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b75036cdf9e04ea4aa767fd12016f177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.232033 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 77c77a11ef384709aef5b62273380573 (DEFAULT_TABLE table=default.Table_a9f63e6f47cb4390b4fff24fd5d4f83e [id=2edd1164c07a4a67b51262d402d53596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.251451 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c77a11ef384709aef5b62273380573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.251832 28468 tablet_service.cc:1467] Processing CreateTablet for tablet e4a544fc27624dcc8a4cb003e59408f2 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.232470 28336 tablet_service.cc:1467] Processing CreateTablet for tablet d93fea5caec94695a5df26d3630c50c7 (DEFAULT_TABLE table=default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.240500 28323 tablet_service.cc:1467] Processing CreateTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 (DEFAULT_TABLE table=default.Table_507e44ddf53d4f36bf1c7179093063ff [id=6539aa29cb864a2fa19c8a2f1ac7bd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.252290 28462 tablet_service.cc:1467] Processing CreateTablet for tablet b75036cdf9e04ea4aa767fd12016f177 (DEFAULT_TABLE table=default.Table_55ff9f3cd6f84368add48d9cecc3fb1b [id=30ff854aabe84b2cb03824d287ae5685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.241021 28329 tablet_service.cc:1467] Processing CreateTablet for tablet e4a544fc27624dcc8a4cb003e59408f2 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.253055 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 (DEFAULT_TABLE table=default.Table_78f62674ace541fabf9047f21a7cadbc [id=aed2a71b61da4d3981c77404ba8dffaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.253186 28467 tablet_service.cc:1467] Processing CreateTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a (DEFAULT_TABLE table=default.Table_11703c3db99a47cc85741ba32923f695 [id=7448b8a18a6d45c48bc56d1379672494]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.253198 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 57ab077988364bd687e54f90c3c265d5 (DEFAULT_TABLE table=default.Table_d121667544004d9e83e4daf3f58f7473 [id=7877e3c7f4b14fcebfd7bec2e15b6240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.241497 28334 tablet_service.cc:1467] Processing CreateTablet for tablet a05f786073084bd1b8c472c20dc8bff6 (DEFAULT_TABLE table=default.Table_f010ec576f0a40cb91c71b23da613670 [id=3221ffb751294f5b9f7fa7848d9845e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.241909 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a (DEFAULT_TABLE table=default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.242313 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.245278 28319 tablet_service.cc:1467] Processing CreateTablet for tablet b75036cdf9e04ea4aa767fd12016f177 (DEFAULT_TABLE table=default.Table_55ff9f3cd6f84368add48d9cecc3fb1b [id=30ff854aabe84b2cb03824d287ae5685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.254249 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 07354861ee6142d6b1fbe573397d3160 (DEFAULT_TABLE table=default.Table_ba1e59290eb64bd488b68e68cb9b2548 [id=2cc7a41775044f4a993bd36fc229d152]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.255363 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd405fd84d44076b1792b7fcbe00e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.255460 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 (DEFAULT_TABLE table=default.Table_30902594a17047d0ac5f2784f7a1d2eb [id=7ca3beb0b91840d08d178367a46c9936]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.256423 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ac912a3bd64cdc9082596e09c72e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.256985 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 173) took 3783 ms. Trace:
I20240503 14:12:08.257148 28603 rpcz_store.cc:276] 0503 14:12:04.473150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.473229 (+ 79us) service_pool.cc:225] Handling call
0503 14:12:08.181509 (+3708280us) ts_tablet_manager.cc:1120] Deleting tablet 2ef6cd66f34c4980bad5e500309b1568
0503 14:12:08.181521 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.199029 (+ 17508us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.222089 (+ 23060us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.256893 (+ 34804us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3698689}
I20240503 14:12:08.257826 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 174) took 3784 ms. Trace:
I20240503 14:12:08.257967 28603 rpcz_store.cc:276] 0503 14:12:04.474099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.482908 (+ 8809us) service_pool.cc:225] Handling call
0503 14:12:08.257528 (+3774620us) ts_tablet_manager.cc:1120] Deleting tablet 265dc5f2630e453ea71abddb1a9753f3
0503 14:12:08.257539 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.257710 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3774324}
I20240503 14:12:08.258460 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 175) took 3782 ms. Trace:
I20240503 14:12:08.258581 28603 rpcz_store.cc:276] 0503 14:12:04.476309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.483226 (+ 6917us) service_pool.cc:225] Handling call
0503 14:12:08.258240 (+3775014us) ts_tablet_manager.cc:1120] Deleting tablet 779f373ef12a4da091309fcf247b5693
0503 14:12:08.258249 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.258355 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3769818}
I20240503 14:12:08.258956 28603 tablet_replica.cc:330] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.259208 28603 raft_consensus.cc:2229] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.259534 28603 raft_consensus.cc:2258] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.260418 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.262964 28458 tablet_service.cc:1467] Processing CreateTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 (DEFAULT_TABLE table=default.Table_507e44ddf53d4f36bf1c7179093063ff [id=6539aa29cb864a2fa19c8a2f1ac7bd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.263595 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd405fd84d44076b1792b7fcbe00e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.264462 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ab077988364bd687e54f90c3c265d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.268465 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07354861ee6142d6b1fbe573397d3160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.272629 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a544fc27624dcc8a4cb003e59408f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.272751 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.273214 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8411f474c247c2abfd0bd9682fb8a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.276600 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1958dc488e4854a8c8e74da9fab243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.276602 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 2ef6cd66f34c4980bad5e500309b1568 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]) successfully deleted
W20240503 14:12:08.277345 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 265dc5f2630e453ea71abddb1a9753f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 265dc5f2630e453ea71abddb1a9753f3 already in progress: creating tablet
W20240503 14:12:08.277715 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 779f373ef12a4da091309fcf247b5693 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779f373ef12a4da091309fcf247b5693 already in progress: creating tablet
I20240503 14:12:08.278972 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.279246 28474 tablet_service.cc:1467] Processing CreateTablet for tablet a05f786073084bd1b8c472c20dc8bff6 (DEFAULT_TABLE table=default.Table_f010ec576f0a40cb91c71b23da613670 [id=3221ffb751294f5b9f7fa7848d9845e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.279794 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05f786073084bd1b8c472c20dc8bff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.281539 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b75036cdf9e04ea4aa767fd12016f177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.283612 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b176a071ba4f3fbff2a64d6e46027a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.287847 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d3937b734026471d8da300845379e2dd" } modify_external_catalogs: true
I20240503 14:12:08.288913 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d3937b734026471d8da300845379e2dd" } modify_external_catalogs: true
I20240503 14:12:08.292985 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b75036cdf9e04ea4aa767fd12016f177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.293612 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 (DEFAULT_TABLE table=default.Table_78f62674ace541fabf9047f21a7cadbc [id=aed2a71b61da4d3981c77404ba8dffaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.297653 28653 ts_tablet_manager.cc:1888] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 14:12:08.302901 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a (DEFAULT_TABLE table=default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.233067 28472 tablet_service.cc:1467] Processing CreateTablet for tablet d93fea5caec94695a5df26d3630c50c7 (DEFAULT_TABLE table=default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.233547 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 77c77a11ef384709aef5b62273380573 (DEFAULT_TABLE table=default.Table_a9f63e6f47cb4390b4fff24fd5d4f83e [id=2edd1164c07a4a67b51262d402d53596]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.303501 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b176a071ba4f3fbff2a64d6e46027a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.297540 28186 tablet_service.cc:1467] Processing CreateTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a (DEFAULT_TABLE table=default.Table_11703c3db99a47cc85741ba32923f695 [id=7448b8a18a6d45c48bc56d1379672494]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.303877 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93fea5caec94695a5df26d3630c50c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.302913 29153 raft_consensus.cc:482] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.304620 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 57ab077988364bd687e54f90c3c265d5 (DEFAULT_TABLE table=default.Table_d121667544004d9e83e4daf3f58f7473 [id=7877e3c7f4b14fcebfd7bec2e15b6240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.309119 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 (DEFAULT_TABLE table=default.Table_78f62674ace541fabf9047f21a7cadbc [id=aed2a71b61da4d3981c77404ba8dffaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.311228 28653 log.cc:1196] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/8bfc56c4591c437da5b9686b726686b2
I20240503 14:12:08.311641 28318 tablet_service.cc:1467] Processing CreateTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a (DEFAULT_TABLE table=default.Table_11703c3db99a47cc85741ba32923f695 [id=7448b8a18a6d45c48bc56d1379672494]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.312379 28653 ts_tablet_manager.cc:1909] T 8bfc56c4591c437da5b9686b726686b2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.312417 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a544fc27624dcc8a4cb003e59408f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.312845 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 57ab077988364bd687e54f90c3c265d5 (DEFAULT_TABLE table=default.Table_d121667544004d9e83e4daf3f58f7473 [id=7877e3c7f4b14fcebfd7bec2e15b6240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.313903 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ac912a3bd64cdc9082596e09c72e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.315985 29113 raft_consensus.cc:482] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.315953 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 (DEFAULT_TABLE table=default.Table_30902594a17047d0ac5f2784f7a1d2eb [id=7ca3beb0b91840d08d178367a46c9936]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.317216 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.318197 29113 raft_consensus.cc:504] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.318221 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.319502 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 07354861ee6142d6b1fbe573397d3160 (DEFAULT_TABLE table=default.Table_ba1e59290eb64bd488b68e68cb9b2548 [id=2cc7a41775044f4a993bd36fc229d152]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.320147 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ab077988364bd687e54f90c3c265d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.320979 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 07354861ee6142d6b1fbe573397d3160 (DEFAULT_TABLE table=default.Table_ba1e59290eb64bd488b68e68cb9b2548 [id=2cc7a41775044f4a993bd36fc229d152]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.321574 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ab077988364bd687e54f90c3c265d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.321581 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c77a11ef384709aef5b62273380573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.322052 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 180) took 2909 ms. Trace:
I20240503 14:12:08.322316 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07354861ee6142d6b1fbe573397d3160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.327028 29107 raft_consensus.cc:482] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.328434 29107 raft_consensus.cc:504] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.322185 28653 rpcz_store.cc:276] 0503 14:12:05.407389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.407919 (+ 530us) service_pool.cc:225] Handling call
0503 14:12:08.179458 (+2771539us) ts_tablet_manager.cc:1120] Deleting tablet 8bfc56c4591c437da5b9686b726686b2
0503 14:12:08.179472 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.222946 (+ 43474us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.290279 (+ 67333us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.316690 (+ 26411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2771201}
I20240503 14:12:08.329372 29107 leader_election.cc:288] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:08.329742 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 (DEFAULT_TABLE table=default.Table_30902594a17047d0ac5f2784f7a1d2eb [id=7ca3beb0b91840d08d178367a46c9936]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:08.330430 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07354861ee6142d6b1fbe573397d3160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.330580 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_955617b294c94c299632de62b5973057 [id=3a24797cc53b439da44cc77025fa9f92]) successfully deleted
I20240503 14:12:08.333498 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:08.336664 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.337139 28653 tablet_replica.cc:330] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:08.340217 28150 leader_election.cc:341] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.343617 29113 leader_election.cc:288] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:08.344727 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.327140 29153 raft_consensus.cc:504] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.345309 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.346228 29153 leader_election.cc:288] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:08.346520 28653 raft_consensus.cc:2229] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:08.346962 28150 leader_election.cc:341] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.347030 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ac912a3bd64cdc9082596e09c72e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.347122 28150 leader_election.cc:302] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:08.354558 29113 raft_consensus.cc:2726] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:08.357133 28427 leader_election.cc:341] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a8d2600031df4d77b9afe636a9b5ad40
I20240503 14:12:08.360368 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05f786073084bd1b8c472c20dc8bff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.360411 28653 raft_consensus.cc:2258] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.365217 28653 ts_tablet_manager.cc:1875] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.369433 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93fea5caec94695a5df26d3630c50c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:08.376283 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8411f474c247c2abfd0bd9682fb8a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:08.385057 28290 leader_election.cc:341] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: ea3e9c030a424740bb2fb641328adfe1
W20240503 14:12:08.394002 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 45590us
I20240503 14:12:08.408663 29153 consensus_queue.cc:1035] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.422982 28600 tablet_bootstrap.cc:492] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.423216 28600 ts_tablet_manager.cc:1367] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.216s user 0.005s sys 0.006s
I20240503 14:12:08.224699 28604 ts_tablet_manager.cc:1367] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.228s user 0.006s sys 0.002s
I20240503 14:12:08.424594 28600 raft_consensus.cc:348] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.424882 28600 raft_consensus.cc:374] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.425016 28600 raft_consensus.cc:729] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.425436 28600 consensus_queue.cc:260] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.426192 28600 ts_tablet_manager.cc:1398] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:08.426595 28600 tablet_bootstrap.cc:492] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.429279 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:08.429543 28600 tablet_bootstrap.cc:654] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.430583 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2f89fca0480d4bf28160384ae0421bce
I20240503 14:12:08.431236 28031 catalog_manager.cc:6554] ReplaceTablet: tablet 2f89fca0480d4bf28160384ae0421bce deleted and replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:08.433936 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6c7c5dfac02d4dfe932996d2a37fe031
I20240503 14:12:08.434588 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 6c7c5dfac02d4dfe932996d2a37fe031 deleted and replaced by tablet 02d45788ae644a8581acca54cc55dd3f
I20240503 14:12:08.434998 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_327bf3afee7847bd9497b8340b940508" } modify_external_catalogs: true
I20240503 14:12:08.435249 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_327bf3afee7847bd9497b8340b940508" } modify_external_catalogs: true
I20240503 14:12:08.437072 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1751f775fe6943d9b5fcc3b113393a73
I20240503 14:12:08.441044 28661 ts_tablet_manager.cc:1875] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.442750 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f89fca0480d4bf28160384ae0421bce with delete_type TABLET_DATA_DELETED (replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.445463 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f89fca0480d4bf28160384ae0421bce with delete_type TABLET_DATA_DELETED (replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.447007 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d45788ae644a8581acca54cc55dd3f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.448068 28604 raft_consensus.cc:348] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.448585 28604 raft_consensus.cc:374] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.448760 28604 raft_consensus.cc:729] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.449247 28604 consensus_queue.cc:260] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.453778 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9d9d27627b064475a2f86b5ddcf4236f" } modify_external_catalogs: true
I20240503 14:12:08.454243 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9d9d27627b064475a2f86b5ddcf4236f" } modify_external_catalogs: true
I20240503 14:12:08.454713 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_28c16c4a952d45c0b34fd1d4864ac748"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.456730 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28c16c4a952d45c0b34fd1d4864ac748 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.458627 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d57cf3dd2d394751bf1d54730635a8dd" } new_table_name: "default.Table_85ee5bde7ad044389efb107eefd39fd3" modify_external_catalogs: true
I20240503 14:12:08.458933 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d45788ae644a8581acca54cc55dd3f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.460742 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a1315a4233714b3f81e5a926c45e14c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.461494 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1315a4233714b3f81e5a926c45e14c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.463388 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_89068da455a94e24972d9924fdab0795" } new_table_name: "default.Table_28594db96b8b48b9af52ccdd7879ff83" modify_external_catalogs: true
I20240503 14:12:08.245930 28614 tablet_bootstrap.cc:492] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:08.468359 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6d6b50aed01a4bf8af8889036dd83606
I20240503 14:12:08.469363 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 903f11a608b1483887d9b5234a404470
I20240503 14:12:08.469727 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_98de9a28340945fc977e41d46d3361c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.470826 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98de9a28340945fc977e41d46d3361c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.466171 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 1751f775fe6943d9b5fcc3b113393a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.472265 29153 raft_consensus.cc:482] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.472481 29153 raft_consensus.cc:504] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.470083 28030 catalog_manager.cc:6554] ReplaceTablet: tablet 903f11a608b1483887d9b5234a404470 deleted and replaced by tablet 3e7b2865e9604ce58280321c7ee2a88e
I20240503 14:12:08.468488 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet bf21f6234dd64cf8a32f1599fe50746e
I20240503 14:12:08.473315 29153 leader_election.cc:288] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:08.473547 28029 catalog_manager.cc:6554] ReplaceTablet: tablet bf21f6234dd64cf8a32f1599fe50746e deleted and replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df
I20240503 14:12:08.474022 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_db4631ebdf2b4a0498017f281e91a22e" } new_table_name: "default.Table_e19b96f79a6c41c0bc266f1391941564" modify_external_catalogs: true
I20240503 14:12:08.472476 28604 ts_tablet_manager.cc:1398] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.049s user 0.003s sys 0.000s
I20240503 14:12:08.478545 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5992e28f1745678618230c183609e4" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:08.479053 28217 raft_consensus.cc:2454] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:08.479671 28427 leader_election.cc:302] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:08.480053 29030 raft_consensus.cc:2781] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.480209 29030 raft_consensus.cc:482] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.480368 29030 raft_consensus.cc:3037] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.481122 28604 tablet_bootstrap.cc:492] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:08.483999 28603 ts_tablet_manager.cc:1875] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:08.485008 28290 leader_election.cc:341] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: ea3e9c030a424740bb2fb641328adfe1
I20240503 14:12:08.483990 29030 raft_consensus.cc:504] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.486075 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d2600031df4d77b9afe636a9b5ad40" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.487262 28357 raft_consensus.cc:2454] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:08.487627 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5992e28f1745678618230c183609e4" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:08.488044 28217 raft_consensus.cc:3037] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.488850 28290 leader_election.cc:302] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:08.494784 28614 ts_tablet_manager.cc:1367] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.281s user 0.001s sys 0.008s
I20240503 14:12:08.496135 29107 raft_consensus.cc:2726] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:08.496567 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 89671.1us
I20240503 14:12:08.512037 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5992e28f1745678618230c183609e4" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.512682 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5992e28f1745678618230c183609e4" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:08.514766 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.516023 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3e9c030a424740bb2fb641328adfe1 with delete_type TABLET_DATA_DELETED (replaced by tablet bd76060daab84c2f9f6727b8a9bb3a49) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.520040 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet bf21f6234dd64cf8a32f1599fe50746e with delete_type TABLET_DATA_DELETED (replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.520051 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 903f11a608b1483887d9b5234a404470 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e7b2865e9604ce58280321c7ee2a88e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.498291 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet f45eb624434a404d85dcd233721ebf68 with delete_type TABLET_DATA_DELETED (replaced by tablet 5775006e9f8f4612aa70dfb316de9ea1) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.520375 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.492050 28217 raft_consensus.cc:2454] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:08.524365 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d45788ae644a8581acca54cc55dd3f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.524616 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f89fca0480d4bf28160384ae0421bce with delete_type TABLET_DATA_DELETED (replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.526899 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 1751f775fe6943d9b5fcc3b113393a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.530263 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.463827 28653 ts_tablet_manager.cc:1888] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.464866 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1751f775fe6943d9b5fcc3b113393a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.536444 28653 log.cc:1196] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/fde0e59d911d4c1eb5125b1f869b396c
I20240503 14:12:08.537532 28653 ts_tablet_manager.cc:1909] T fde0e59d911d4c1eb5125b1f869b396c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.538329 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 903f11a608b1483887d9b5234a404470 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e7b2865e9604ce58280321c7ee2a88e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.539141 29178 consensus_queue.cc:1035] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:12:08.540426 28614 raft_consensus.cc:348] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.541838 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 181) took 3069 ms. Trace:
I20240503 14:12:08.544319 28653 rpcz_store.cc:276] 0503 14:12:05.471915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.472001 (+ 86us) service_pool.cc:225] Handling call
0503 14:12:08.329366 (+2857365us) ts_tablet_manager.cc:1120] Deleting tablet fde0e59d911d4c1eb5125b1f869b396c
0503 14:12:08.329381 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.407496 (+ 78115us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.463576 (+ 56080us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.541788 (+ 78212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2856998}
I20240503 14:12:08.544731 28600 tablet_bootstrap.cc:492] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.544811 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.545017 28600 ts_tablet_manager.cc:1367] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.118s user 0.049s sys 0.049s
I20240503 14:12:08.545025 29030 leader_election.cc:288] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:08.545500 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 186) took 2888 ms. Trace:
I20240503 14:12:08.545712 28653 rpcz_store.cc:276] 0503 14:12:05.657033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.657101 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:08.545203 (+2888102us) ts_tablet_manager.cc:1120] Deleting tablet 1044f34588d045ce898435b61a90503f
0503 14:12:08.545214 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.545439 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2887316}
I20240503 14:12:08.546439 28653 tablet_replica.cc:330] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.546797 28653 raft_consensus.cc:2229] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.547081 28653 pending_rounds.cc:62] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:08.547291 28653 pending_rounds.cc:68] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597681116725248 op_type: NO_OP noop_request { }
I20240503 14:12:08.547564 28653 raft_consensus.cc:2866] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:08.546301 28600 raft_consensus.cc:348] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.549227 28427 leader_election.cc:302] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:08.549507 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet bf21f6234dd64cf8a32f1599fe50746e with delete_type TABLET_DATA_DELETED (replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.549753 28427 leader_election.cc:302] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:08.554702 28603 ts_tablet_manager.cc:1888] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.555393 28600 raft_consensus.cc:374] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.555590 28600 raft_consensus.cc:729] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.556077 28600 consensus_queue.cc:260] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.556898 28653 raft_consensus.cc:2258] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.558187 28600 ts_tablet_manager.cc:1398] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240503 14:12:08.558485 28603 log.cc:1196] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/b1de9717d0e44179b875174d5a76e4c3
I20240503 14:12:08.558651 28600 tablet_bootstrap.cc:492] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.559087 28603 ts_tablet_manager.cc:1909] T b1de9717d0e44179b875174d5a76e4c3 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.560009 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 176) took 4077 ms. Trace:
I20240503 14:12:08.561159 28600 tablet_bootstrap.cc:654] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.561352 28603 rpcz_store.cc:276] 0503 14:12:04.482585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:04.488487 (+ 5902us) service_pool.cc:225] Handling call
0503 14:12:08.258793 (+3770306us) ts_tablet_manager.cc:1120] Deleting tablet b1de9717d0e44179b875174d5a76e4c3
0503 14:12:08.258802 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.546715 (+287913us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.554523 (+ 7808us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.559961 (+ 5438us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3766812}
I20240503 14:12:08.562291 28653 ts_tablet_manager.cc:1875] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.562855 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 178) took 3434 ms. Trace:
I20240503 14:12:08.563040 28603 rpcz_store.cc:276] 0503 14:12:05.128560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.129646 (+ 1086us) service_pool.cc:225] Handling call
0503 14:12:08.562630 (+3432984us) ts_tablet_manager.cc:1120] Deleting tablet a6dbb59d98834ab6bbac4abfa92681e3
0503 14:12:08.562645 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.562799 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3420220}
I20240503 14:12:08.568598 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 179) took 3419 ms. Trace:
I20240503 14:12:08.568984 28603 rpcz_store.cc:276] 0503 14:12:05.148905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.149293 (+ 388us) service_pool.cc:225] Handling call
0503 14:12:08.568330 (+3419037us) ts_tablet_manager.cc:1120] Deleting tablet ed9262a7e828489ab51a745068c737df
0503 14:12:08.568345 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.568526 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3418739}
I20240503 14:12:08.569586 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 180) took 3420 ms. Trace:
I20240503 14:12:08.569836 28603 rpcz_store.cc:276] 0503 14:12:05.149239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.149659 (+ 420us) service_pool.cc:225] Handling call
0503 14:12:08.569372 (+3419713us) ts_tablet_manager.cc:1120] Deleting tablet 5d637087c60144aaaa9b0965110d280e
0503 14:12:08.569384 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.569527 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3419493}
I20240503 14:12:08.570291 28603 tablet_replica.cc:330] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.570593 28603 raft_consensus.cc:2229] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.570854 28603 raft_consensus.cc:2258] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.571936 28603 ts_tablet_manager.cc:1875] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.574829 28653 ts_tablet_manager.cc:1888] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.575045 28653 log.cc:1196] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/1693dd36a3d84d88a49690c537590083
I20240503 14:12:08.575583 28653 ts_tablet_manager.cc:1909] T 1693dd36a3d84d88a49690c537590083 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.589355 28600 tablet_bootstrap.cc:492] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.589576 28600 ts_tablet_manager.cc:1367] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20240503 14:12:08.590647 28600 raft_consensus.cc:348] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.590983 28600 raft_consensus.cc:374] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.591130 28600 raft_consensus.cc:729] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.591465 28600 consensus_queue.cc:260] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.592931 28603 ts_tablet_manager.cc:1888] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.593066 29107 consensus_queue.cc:1035] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.593222 28603 log.cc:1196] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/f45eb624434a404d85dcd233721ebf68
I20240503 14:12:08.593730 28603 ts_tablet_manager.cc:1909] T f45eb624434a404d85dcd233721ebf68 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.594734 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 181) took 3334 ms. Trace:
I20240503 14:12:08.594969 28603 rpcz_store.cc:276] 0503 14:12:05.259855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.259919 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:08.570195 (+3310276us) ts_tablet_manager.cc:1120] Deleting tablet f45eb624434a404d85dcd233721ebf68
0503 14:12:08.570204 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.588858 (+ 18654us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.592776 (+ 3918us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.594666 (+ 1890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309916}
I20240503 14:12:08.595760 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 182) took 3334 ms. Trace:
I20240503 14:12:08.595973 28603 rpcz_store.cc:276] 0503 14:12:05.261312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.261368 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:08.595453 (+3334085us) ts_tablet_manager.cc:1120] Deleting tablet bd76060daab84c2f9f6727b8a9bb3a49
0503 14:12:08.595466 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.595662 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3333794}
I20240503 14:12:08.596288 28600 ts_tablet_manager.cc:1398] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 14:12:08.596689 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 183) took 3334 ms. Trace:
I20240503 14:12:08.596896 28603 rpcz_store.cc:276] 0503 14:12:05.261758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.261804 (+ 46us) service_pool.cc:225] Handling call
0503 14:12:08.596457 (+3334653us) ts_tablet_manager.cc:1120] Deleting tablet 63e977f7837e49fb93ef0fd18b619792
0503 14:12:08.596468 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.596597 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3334399}
I20240503 14:12:08.597465 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 184) took 3191 ms. Trace:
I20240503 14:12:08.597652 28603 rpcz_store.cc:276] 0503 14:12:05.406387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.406452 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:08.597231 (+3190779us) ts_tablet_manager.cc:1120] Deleting tablet 7c6f41bb9c5d40b0913264864ecc3528
0503 14:12:08.597241 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.597359 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3190485}
I20240503 14:12:08.598131 28603 tablet_replica.cc:330] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.598367 28614 raft_consensus.cc:374] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.598925 28614 raft_consensus.cc:729] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.599963 28614 consensus_queue.cc:260] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.600847 28600 tablet_bootstrap.cc:492] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.607298 28614 ts_tablet_manager.cc:1398] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.112s user 0.000s sys 0.003s
I20240503 14:12:08.598456 28603 raft_consensus.cc:2229] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.607911 28614 tablet_bootstrap.cc:492] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:08.607964 28603 raft_consensus.cc:2258] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.608745 28603 ts_tablet_manager.cc:1875] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.608937 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 187) took 2950 ms. Trace:
I20240503 14:12:08.609061 28653 rpcz_store.cc:276] 0503 14:12:05.658697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.658763 (+ 66us) service_pool.cc:225] Handling call
0503 14:12:08.546221 (+2887458us) ts_tablet_manager.cc:1120] Deleting tablet 1693dd36a3d84d88a49690c537590083
0503 14:12:08.546231 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.570827 (+ 24596us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.574625 (+ 3798us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.608887 (+ 34262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2887178}
I20240503 14:12:08.610159 28614 tablet_bootstrap.cc:654] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.611196 28600 tablet_bootstrap.cc:654] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.612560 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet b1de9717d0e44179b875174d5a76e4c3 (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]) successfully deleted
I20240503 14:12:08.612654 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 188) took 2927 ms. Trace:
I20240503 14:12:08.612833 28653 rpcz_store.cc:276] 0503 14:12:05.684919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.684991 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:08.612424 (+2927433us) ts_tablet_manager.cc:1120] Deleting tablet 129717ef137c41c5a08d401c64164187
0503 14:12:08.612436 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.612592 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927075}
W20240503 14:12:08.612937 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a6dbb59d98834ab6bbac4abfa92681e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6dbb59d98834ab6bbac4abfa92681e3 already in progress: creating tablet
W20240503 14:12:08.613252 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ed9262a7e828489ab51a745068c737df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9262a7e828489ab51a745068c737df already in progress: creating tablet
W20240503 14:12:08.613610 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5d637087c60144aaaa9b0965110d280e because the tablet was not found. No further retry: Not found: Tablet not found: 5d637087c60144aaaa9b0965110d280e
I20240503 14:12:08.613914 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet fde0e59d911d4c1eb5125b1f869b396c (table default.Table_666a0b3b2afb4a9fb91fb970e6c7a173 [id=a66a34594c864f0c81ba2da0a1449aee]) successfully deleted
W20240503 14:12:08.614228 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 1044f34588d045ce898435b61a90503f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1044f34588d045ce898435b61a90503f already in progress: creating tablet
I20240503 14:12:08.614506 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 1693dd36a3d84d88a49690c537590083 (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
I20240503 14:12:08.614985 28653 tablet_replica.cc:330] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.617960 28603 ts_tablet_manager.cc:1888] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 14:12:08.618304 28603 log.cc:1196] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/8bfc56c4591c437da5b9686b726686b2
I20240503 14:12:08.618944 28603 ts_tablet_manager.cc:1909] T 8bfc56c4591c437da5b9686b726686b2 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.619603 28653 raft_consensus.cc:2229] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.620697 28653 raft_consensus.cc:2258] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.621469 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 185) took 3214 ms. Trace:
I20240503 14:12:08.625221 28604 tablet_bootstrap.cc:654] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.625456 28603 rpcz_store.cc:276] 0503 14:12:05.407383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.407442 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:08.597995 (+3190553us) ts_tablet_manager.cc:1120] Deleting tablet 8bfc56c4591c437da5b9686b726686b2
0503 14:12:08.598005 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.613223 (+ 15218us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.617768 (+ 4545us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.621386 (+ 3618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3190020}
I20240503 14:12:08.626127 28603 tablet_replica.cc:330] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.626534 28603 raft_consensus.cc:2229] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.626847 28603 raft_consensus.cc:2258] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.627714 28653 ts_tablet_manager.cc:1875] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.627804 28603 ts_tablet_manager.cc:1875] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.635576 28653 ts_tablet_manager.cc:1888] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.635823 28653 log.cc:1196] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f51834695b8b4a729a088da81eee2e47
I20240503 14:12:08.638375 29107 consensus_queue.cc:1035] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.638947 28661 ts_tablet_manager.cc:1888] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:08.648396 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 129717ef137c41c5a08d401c64164187 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 129717ef137c41c5a08d401c64164187 already in progress: creating tablet
I20240503 14:12:08.650869 28032 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 7ebdb5395b944261bf7044a48bb0b55c
I20240503 14:12:08.652730 28603 ts_tablet_manager.cc:1888] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.653061 28661 log.cc:1196] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/f45eb624434a404d85dcd233721ebf68
W20240503 14:12:08.653439 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b90e9563b4c4cef8ee7964eaa96e285 consensus state has no leader
I20240503 14:12:08.653753 28661 ts_tablet_manager.cc:1909] T f45eb624434a404d85dcd233721ebf68 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:08.654778 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 172) took 3395 ms. Trace:
I20240503 14:12:08.655197 28603 log.cc:1196] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/fde0e59d911d4c1eb5125b1f869b396c
I20240503 14:12:08.655745 28603 ts_tablet_manager.cc:1909] T fde0e59d911d4c1eb5125b1f869b396c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.656759 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 187) took 3181 ms. Trace:
I20240503 14:12:08.656999 28661 rpcz_store.cc:276] 0503 14:12:05.259707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.259770 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:08.165665 (+2905895us) ts_tablet_manager.cc:1120] Deleting tablet f45eb624434a404d85dcd233721ebf68
0503 14:12:08.165677 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.631835 (+466158us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.638752 (+ 6917us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.654735 (+ 15983us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2546,"tablet-delete.queue_time_us":2905694}
I20240503 14:12:08.657678 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 173) took 3396 ms. Trace:
I20240503 14:12:08.657900 28661 rpcz_store.cc:276] 0503 14:12:05.261689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.261748 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:08.657438 (+3395690us) ts_tablet_manager.cc:1120] Deleting tablet bd76060daab84c2f9f6727b8a9bb3a49
0503 14:12:08.657449 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.657608 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3395441}
I20240503 14:12:08.658187 28603 rpcz_store.cc:276] 0503 14:12:05.474897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.474983 (+ 86us) service_pool.cc:225] Handling call
0503 14:12:08.626016 (+3151033us) ts_tablet_manager.cc:1120] Deleting tablet fde0e59d911d4c1eb5125b1f869b396c
0503 14:12:08.626030 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.631659 (+ 5629us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.652579 (+ 20920us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.656695 (+ 4116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3103615}
I20240503 14:12:08.658767 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 174) took 3395 ms. Trace:
I20240503 14:12:08.659262 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 193) took 2999 ms. Trace:
I20240503 14:12:08.659641 28661 rpcz_store.cc:276] 0503 14:12:05.263237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.263297 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:08.658577 (+3395280us) ts_tablet_manager.cc:1120] Deleting tablet 63e977f7837e49fb93ef0fd18b619792
0503 14:12:08.658588 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.658712 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3395052}
I20240503 14:12:08.659965 28603 rpcz_store.cc:276] 0503 14:12:05.659518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.659636 (+ 118us) service_pool.cc:225] Handling call
0503 14:12:08.659022 (+2999386us) ts_tablet_manager.cc:1120] Deleting tablet 1693dd36a3d84d88a49690c537590083
0503 14:12:08.659035 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.659163 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2999059}
I20240503 14:12:08.660559 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 175) took 3253 ms. Trace:
I20240503 14:12:08.661033 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 192) took 3003 ms. Trace:
I20240503 14:12:08.661435 28661 rpcz_store.cc:276] 0503 14:12:05.407298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.407366 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:08.660342 (+3252976us) ts_tablet_manager.cc:1120] Deleting tablet 7c6f41bb9c5d40b0913264864ecc3528
0503 14:12:08.660352 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.660500 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252477}
I20240503 14:12:08.661720 28603 rpcz_store.cc:276] 0503 14:12:05.657416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.660644 (+ 3228us) service_pool.cc:225] Handling call
0503 14:12:08.660826 (+3000182us) ts_tablet_manager.cc:1120] Deleting tablet 1044f34588d045ce898435b61a90503f
0503 14:12:08.660836 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.660950 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2988666}
I20240503 14:12:08.661929 29186 raft_consensus.cc:2781] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.662178 29186 raft_consensus.cc:482] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.662411 29186 raft_consensus.cc:3037] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.662592 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 195) took 2954 ms. Trace:
I20240503 14:12:08.662220 28661 tablet_replica.cc:330] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:08.663182 28661 raft_consensus.cc:2229] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.663358 28603 rpcz_store.cc:276] 0503 14:12:05.708620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.708701 (+ 81us) service_pool.cc:225] Handling call
0503 14:12:08.662377 (+2953676us) ts_tablet_manager.cc:1120] Deleting tablet f51834695b8b4a729a088da81eee2e47
0503 14:12:08.662389 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.662507 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2953382}
I20240503 14:12:08.663817 28661 raft_consensus.cc:2258] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.664448 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 194) took 2982 ms. Trace:
I20240503 14:12:08.665205 28661 ts_tablet_manager.cc:1875] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:08.665302 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed4dc732bcb44222b4b15e5d77defe33 consensus state has no leader
I20240503 14:12:08.665383 28603 rpcz_store.cc:276] 0503 14:12:05.682408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.711227 (+ 28819us) service_pool.cc:225] Handling call
0503 14:12:08.664037 (+2952810us) ts_tablet_manager.cc:1120] Deleting tablet 129717ef137c41c5a08d401c64164187
0503 14:12:08.664048 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.664369 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2944678}
I20240503 14:12:08.665882 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 196) took 2933 ms. Trace:
W20240503 14:12:08.666445 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60206e8e73e84e409ed6aba222d9ff4b consensus state has no leader
I20240503 14:12:08.667246 28603 rpcz_store.cc:276] 0503 14:12:05.732458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.744059 (+ 11601us) service_pool.cc:225] Handling call
0503 14:12:08.665670 (+2921611us) ts_tablet_manager.cc:1120] Deleting tablet 47ba571a81eb4240a4a4a4a08cb879c5
0503 14:12:08.665680 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.665794 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2921230}
I20240503 14:12:08.667877 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 212) took 2781 ms. Trace:
I20240503 14:12:08.667996 28603 rpcz_store.cc:276] 0503 14:12:05.886308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.886646 (+ 338us) service_pool.cc:225] Handling call
0503 14:12:08.667563 (+2780917us) ts_tablet_manager.cc:1120] Deleting tablet f50ad403bd354559b2522a76af1fa722
0503 14:12:08.667574 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.667792 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774299}
I20240503 14:12:08.668463 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 213) took 2670 ms. Trace:
I20240503 14:12:08.668871 28653 ts_tablet_manager.cc:1909] T f51834695b8b4a729a088da81eee2e47 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.671213 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3169e805aa1543889ee0dab4758ae9e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.672013 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3169e805aa1543889ee0dab4758ae9e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.672597 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_bbc0f1621f4842c2a8b5913acd913703" } new_table_name: "default.Table_69e2f4400fe2428ea429f0d666b9610c" modify_external_catalogs: true
I20240503 14:12:08.673017 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 189) took 2962 ms. Trace:
I20240503 14:12:08.673164 28653 rpcz_store.cc:276] 0503 14:12:05.710086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.710157 (+ 71us) service_pool.cc:225] Handling call
0503 14:12:08.614862 (+2904705us) ts_tablet_manager.cc:1120] Deleting tablet f51834695b8b4a729a088da81eee2e47
0503 14:12:08.614877 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.631603 (+ 16726us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.635402 (+ 3799us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.672886 (+ 37484us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2904411}
I20240503 14:12:08.676214 28603 rpcz_store.cc:276] 0503 14:12:05.998171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.998286 (+ 115us) service_pool.cc:225] Handling call
0503 14:12:08.668262 (+2669976us) ts_tablet_manager.cc:1120] Deleting tablet 92c800e997e944158f7c839dd0e00d44
0503 14:12:08.668270 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.668379 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654686}
I20240503 14:12:08.676658 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_67ce7228ef934797ba768f947d2cb0aa" } new_table_name: "default.Table_ea4ed2f332dc46aaa1f6e7025a1a1c3c" modify_external_catalogs: true
I20240503 14:12:08.677242 28603 tablet_replica.cc:330] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.680578 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 190) took 2939 ms. Trace:
I20240503 14:12:08.680862 28653 rpcz_store.cc:276] 0503 14:12:05.741240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.741731 (+ 491us) service_pool.cc:225] Handling call
0503 14:12:08.680251 (+2938520us) ts_tablet_manager.cc:1120] Deleting tablet 47ba571a81eb4240a4a4a4a08cb879c5
0503 14:12:08.680269 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.680478 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2936424}
I20240503 14:12:08.681711 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 205) took 2791 ms. Trace:
I20240503 14:12:08.681977 28653 rpcz_store.cc:276] 0503 14:12:05.890478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.890547 (+ 69us) service_pool.cc:225] Handling call
0503 14:12:08.681460 (+2790913us) ts_tablet_manager.cc:1120] Deleting tablet f50ad403bd354559b2522a76af1fa722
0503 14:12:08.681472 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.681615 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2789801}
I20240503 14:12:08.683228 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a1bf5785458b47d0a18f3f3088449ea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:08.683387 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 206) took 2745 ms. Trace:
I20240503 14:12:08.683521 28653 rpcz_store.cc:276] 0503 14:12:05.938352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.938430 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:08.683154 (+2744724us) ts_tablet_manager.cc:1120] Deleting tablet 37dc7d96357743089372baf71654b1e3
0503 14:12:08.683162 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.683300 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2738817}
I20240503 14:12:08.687875 28653 tablet_replica.cc:330] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.688344 28653 raft_consensus.cc:2229] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:08.690928 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1bf5785458b47d0a18f3f3088449ea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.692516 28653 raft_consensus.cc:2258] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.692570 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_68199a1d18244f54bb0b7b0256a58e2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.693361 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68199a1d18244f54bb0b7b0256a58e2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.693413 28653 ts_tablet_manager.cc:1875] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.695636 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4fdbe31ba57a42b993d60fa19081c044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.696516 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fdbe31ba57a42b993d60fa19081c044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.697952 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b7d4e4a1f0b14406be957426a45bf45d
I20240503 14:12:08.700762 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ff5facc5efa54652826a5f626e9aca85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:08.701982 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff5facc5efa54652826a5f626e9aca85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:08.709149 29107 consensus_queue.cc:1035] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.709663 28653 ts_tablet_manager.cc:1888] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.716256 28653 log.cc:1196] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ffbd3291c56b4c13813c80d6775afe6f
I20240503 14:12:08.716982 28653 ts_tablet_manager.cc:1909] T ffbd3291c56b4c13813c80d6775afe6f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:08.721400 28561 log.cc:924] Time spent T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240503 14:12:08.721668 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 207) took 2737 ms. Trace:
I20240503 14:12:08.722035 28653 rpcz_store.cc:276] 0503 14:12:05.984572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.984651 (+ 79us) service_pool.cc:225] Handling call
0503 14:12:08.687726 (+2703075us) ts_tablet_manager.cc:1120] Deleting tablet ffbd3291c56b4c13813c80d6775afe6f
0503 14:12:08.687737 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.697033 (+ 9296us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.709455 (+ 12422us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.721612 (+ 12157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2702835}
I20240503 14:12:08.723743 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 214) took 2669 ms. Trace:
I20240503 14:12:08.724051 28653 rpcz_store.cc:276] 0503 14:12:06.053722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.053790 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:08.722708 (+2668918us) ts_tablet_manager.cc:1120] Deleting tablet 0b255d3e3dc64fb69c2ef6ed389db6a6
0503 14:12:08.722722 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.722874 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2667662}
I20240503 14:12:08.724885 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 215) took 2668 ms. Trace:
I20240503 14:12:08.725133 28653 rpcz_store.cc:276] 0503 14:12:06.056040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.056115 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:08.724666 (+2668551us) ts_tablet_manager.cc:1120] Deleting tablet 5775006e9f8f4612aa70dfb316de9ea1
0503 14:12:08.724677 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.724825 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2666462}
W20240503 14:12:08.727291 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 177526us
W20240503 14:12:08.730338 28427 leader_election.cc:341] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:08.732473 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 83646.3us
W20240503 14:12:08.734287 28427 leader_election.cc:341] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.735733 28600 tablet_bootstrap.cc:492] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.738024 28600 ts_tablet_manager.cc:1367] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.137s user 0.010s sys 0.019s
I20240503 14:12:08.738199 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 229) took 2389 ms. Trace:
I20240503 14:12:08.738436 28653 rpcz_store.cc:276] 0503 14:12:06.348505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.348567 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:08.737952 (+2389385us) ts_tablet_manager.cc:1120] Deleting tablet 06fb79fcf30d485798c64c815e372dc1
0503 14:12:08.737965 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.738144 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2372444}
I20240503 14:12:08.739063 28653 tablet_replica.cc:330] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.739413 28653 raft_consensus.cc:2229] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.739295 28600 raft_consensus.cc:348] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.739701 28653 pending_rounds.cc:62] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:08.739740 28600 raft_consensus.cc:374] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.739858 28653 pending_rounds.cc:68] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597681228292096 op_type: NO_OP noop_request { }
I20240503 14:12:08.739986 28600 raft_consensus.cc:729] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.740098 28653 raft_consensus.cc:2866] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:08.740371 28653 raft_consensus.cc:2258] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.740453 28600 consensus_queue.cc:260] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.740738 28603 raft_consensus.cc:2229] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.741017 28603 raft_consensus.cc:2258] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.741258 28653 ts_tablet_manager.cc:1875] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.741708 28603 ts_tablet_manager.cc:1875] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.742743 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d4e4a1f0b14406be957426a45bf45d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:08.743475 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d4e4a1f0b14406be957426a45bf45d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:08.661242 29183 raft_consensus.cc:2781] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.744781 28600 ts_tablet_manager.cc:1398] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:08.745007 29183 raft_consensus.cc:686] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:08.745462 29183 consensus_queue.cc:237] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:08.748170 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 63e977f7837e49fb93ef0fd18b619792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e977f7837e49fb93ef0fd18b619792 already in progress: creating tablet
I20240503 14:12:08.748531 28600 tablet_bootstrap.cc:492] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.750797 28600 tablet_bootstrap.cc:654] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.752581 28604 tablet_bootstrap.cc:492] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:08.752768 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.752878 28604 ts_tablet_manager.cc:1367] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.272s user 0.005s sys 0.000s
W20240503 14:12:08.753453 28290 consensus_peers.cc:485] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:08.754020 28604 raft_consensus.cc:348] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.754359 28604 raft_consensus.cc:374] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.754514 28604 raft_consensus.cc:729] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.754812 28604 consensus_queue.cc:260] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.755497 28604 ts_tablet_manager.cc:1398] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:08.755858 28604 tablet_bootstrap.cc:492] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:08.756639 28600 tablet_bootstrap.cc:492] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.756889 28600 ts_tablet_manager.cc:1367] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240503 14:12:08.757881 28600 raft_consensus.cc:348] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.758211 28600 raft_consensus.cc:374] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.758385 28600 raft_consensus.cc:729] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.758744 28600 consensus_queue.cc:260] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.761157 28614 tablet_bootstrap.cc:492] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:08.761957 28614 ts_tablet_manager.cc:1367] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.154s user 0.004s sys 0.001s
I20240503 14:12:08.762423 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 903f11a608b1483887d9b5234a404470 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e7b2865e9604ce58280321c7ee2a88e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.763108 28600 ts_tablet_manager.cc:1398] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:08.763278 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet bf21f6234dd64cf8a32f1599fe50746e with delete_type TABLET_DATA_DELETED (replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.766569 28600 tablet_bootstrap.cc:492] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.766695 28661 ts_tablet_manager.cc:1888] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 14:12:08.768188 28661 log.cc:1196] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/8bfc56c4591c437da5b9686b726686b2
I20240503 14:12:08.768857 28661 ts_tablet_manager.cc:1909] T 8bfc56c4591c437da5b9686b726686b2 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:08.768956 28600 tablet_bootstrap.cc:654] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.769421 29113 raft_consensus.cc:482] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.769582 29113 raft_consensus.cc:504] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.770339 29113 leader_election.cc:288] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:08.772178 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7c6f41bb9c5d40b0913264864ecc3528 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6f41bb9c5d40b0913264864ecc3528 already in progress: creating tablet
I20240503 14:12:08.772471 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_955617b294c94c299632de62b5973057 [id=3a24797cc53b439da44cc77025fa9f92]) successfully deleted
I20240503 14:12:08.772668 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 176) took 3362 ms. Trace:
I20240503 14:12:08.772790 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet fde0e59d911d4c1eb5125b1f869b396c (table default.Table_666a0b3b2afb4a9fb91fb970e6c7a173 [id=a66a34594c864f0c81ba2da0a1449aee]) successfully deleted
I20240503 14:12:08.775425 28661 rpcz_store.cc:276] 0503 14:12:05.410509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.410576 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:08.662036 (+3251460us) ts_tablet_manager.cc:1120] Deleting tablet 8bfc56c4591c437da5b9686b726686b2
0503 14:12:08.662046 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.673085 (+ 11039us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.766524 (+ 93439us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.772547 (+ 6023us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3251186}
I20240503 14:12:08.776562 28653 ts_tablet_manager.cc:1888] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.776872 28653 log.cc:1196] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/eee989ff083041dc8dcd2cd1b220780d
I20240503 14:12:08.777549 28653 ts_tablet_manager.cc:1909] T eee989ff083041dc8dcd2cd1b220780d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.771195 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dab0296a0894a63afc16ed466c591d5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.778563 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 230) took 2412 ms. Trace:
W20240503 14:12:08.778733 28150 leader_election.cc:341] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.778972 28653 rpcz_store.cc:276] 0503 14:12:06.366915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.366996 (+ 81us) service_pool.cc:225] Handling call
0503 14:12:08.738964 (+2371968us) ts_tablet_manager.cc:1120] Deleting tablet eee989ff083041dc8dcd2cd1b220780d
0503 14:12:08.738974 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.744972 (+ 5998us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.776368 (+ 31396us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.778516 (+ 2148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2140177}
I20240503 14:12:08.779860 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 231) took 2176 ms. Trace:
I20240503 14:12:08.779726 28614 raft_consensus.cc:348] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.780194 28614 raft_consensus.cc:374] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.780100 28653 rpcz_store.cc:276] 0503 14:12:06.603875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.603966 (+ 91us) service_pool.cc:225] Handling call
0503 14:12:08.779628 (+2175662us) ts_tablet_manager.cc:1120] Deleting tablet 65cdd090bf0f493f8b92341369378532
0503 14:12:08.779640 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.779792 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2085177}
I20240503 14:12:08.780424 28614 raft_consensus.cc:729] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.780620 28653 tablet_replica.cc:330] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.780822 28614 consensus_queue.cc:260] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.780938 28653 raft_consensus.cc:2229] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.781845 28614 ts_tablet_manager.cc:1398] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:08.782250 28653 raft_consensus.cc:2258] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.787094 28653 ts_tablet_manager.cc:1875] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.788419 28661 tablet_replica.cc:330] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:08.788785 28661 raft_consensus.cc:2229] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:08.789211 28661 pending_rounds.cc:62] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:08.775272 28603 ts_tablet_manager.cc:1888] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.789505 28603 log.cc:1196] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/4775ff1029e6449c83219030e3809984
I20240503 14:12:08.790140 28603 ts_tablet_manager.cc:1909] T 4775ff1029e6449c83219030e3809984 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.776885 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d4e4a1f0b14406be957426a45bf45d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:08.790465 28661 pending_rounds.cc:68] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597682780786688 op_type: NO_OP noop_request { }
I20240503 14:12:08.790737 28661 raft_consensus.cc:2866] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:08.790941 28661 raft_consensus.cc:2258] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.792321 28661 ts_tablet_manager.cc:1875] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:08.796706 28290 consensus_peers.cc:485] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0e8b0eed57c4ad1bf6040507f6f8d4c. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:08.797690 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1693dd36a3d84d88a49690c537590083 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1693dd36a3d84d88a49690c537590083 already in progress: creating tablet
W20240503 14:12:08.798043 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1044f34588d045ce898435b61a90503f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1044f34588d045ce898435b61a90503f already in progress: creating tablet
W20240503 14:12:08.798317 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f51834695b8b4a729a088da81eee2e47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51834695b8b4a729a088da81eee2e47 already in progress: creating tablet
W20240503 14:12:08.798619 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 129717ef137c41c5a08d401c64164187 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 129717ef137c41c5a08d401c64164187 already in progress: creating tablet
W20240503 14:12:08.798861 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 47ba571a81eb4240a4a4a4a08cb879c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47ba571a81eb4240a4a4a4a08cb879c5 already in progress: creating tablet
W20240503 14:12:08.799113 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f50ad403bd354559b2522a76af1fa722 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50ad403bd354559b2522a76af1fa722 already in progress: creating tablet
W20240503 14:12:08.799353 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 92c800e997e944158f7c839dd0e00d44 because the tablet was not found. No further retry: Not found: Tablet not found: 92c800e997e944158f7c839dd0e00d44
I20240503 14:12:08.799585 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet f51834695b8b4a729a088da81eee2e47 (table default.Table_b0e228f901e4427ca9bbfc14b02c1f7f [id=743c045f2e104da8b09fd13245037eb6]) successfully deleted
W20240503 14:12:08.799827 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 47ba571a81eb4240a4a4a4a08cb879c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47ba571a81eb4240a4a4a4a08cb879c5 already in progress: creating tablet
I20240503 14:12:08.802385 28661 ts_tablet_manager.cc:1888] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.805131 28661 log.cc:1196] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/fde0e59d911d4c1eb5125b1f869b396c
I20240503 14:12:08.805543 29187 raft_consensus.cc:482] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.805737 29187 raft_consensus.cc:504] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.806859 28600 tablet_bootstrap.cc:492] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.807179 28600 ts_tablet_manager.cc:1367] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.018s
I20240503 14:12:08.807301 29187 leader_election.cc:288] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:08.787526 29186 raft_consensus.cc:504] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.774269 28604 tablet_bootstrap.cc:654] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.775792 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dab0296a0894a63afc16ed466c591d5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:08.808524 28150 leader_election.cc:341] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.808717 28150 leader_election.cc:302] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:08.809029 28427 consensus_peers.cc:485] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1693dd36a3d84d88a49690c537590083. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:08.809159 29113 raft_consensus.cc:2726] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:08.808290 28600 raft_consensus.cc:348] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.809720 28600 raft_consensus.cc:374] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.809857 28600 raft_consensus.cc:729] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.810196 28600 consensus_queue.cc:260] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.812590 29186 leader_election.cc:288] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:08.814908 28600 ts_tablet_manager.cc:1398] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20240503 14:12:08.816398 28427 consensus_peers.cc:485] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:08.818997 28614 tablet_bootstrap.cc:492] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:08.819416 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 214) took 2808 ms. Trace:
I20240503 14:12:08.820188 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:08.824538 28603 rpcz_store.cc:276] 0503 14:12:06.011337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.012177 (+ 840us) service_pool.cc:225] Handling call
0503 14:12:08.677083 (+2664906us) ts_tablet_manager.cc:1120] Deleting tablet 4775ff1029e6449c83219030e3809984
0503 14:12:08.677097 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.745771 (+ 68674us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.775094 (+ 29323us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.819343 (+ 44249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2656255}
I20240503 14:12:08.825045 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 215) took 2805 ms. Trace:
I20240503 14:12:08.825057 28661 ts_tablet_manager.cc:1909] T fde0e59d911d4c1eb5125b1f869b396c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:08.825199 28603 rpcz_store.cc:276] 0503 14:12:06.019636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.019764 (+ 128us) service_pool.cc:225] Handling call
0503 14:12:08.824849 (+2805085us) ts_tablet_manager.cc:1120] Deleting tablet 37dc7d96357743089372baf71654b1e3
0503 14:12:08.824860 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.824989 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2800392}
I20240503 14:12:08.825623 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 217) took 2795 ms. Trace:
I20240503 14:12:08.825793 28603 rpcz_store.cc:276] 0503 14:12:06.029674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.029770 (+ 96us) service_pool.cc:225] Handling call
0503 14:12:08.825462 (+2795692us) ts_tablet_manager.cc:1120] Deleting tablet ffbd3291c56b4c13813c80d6775afe6f
0503 14:12:08.825471 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.825566 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2659884}
I20240503 14:12:08.826038 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 177) took 3351 ms. Trace:
I20240503 14:12:08.826206 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 224) took 2598 ms. Trace:
I20240503 14:12:08.826162 28661 rpcz_store.cc:276] 0503 14:12:05.474748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.474811 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:08.788271 (+3313460us) ts_tablet_manager.cc:1120] Deleting tablet fde0e59d911d4c1eb5125b1f869b396c
0503 14:12:08.788284 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.797438 (+ 9154us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.802230 (+ 4792us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.825978 (+ 23748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3313048}
I20240503 14:12:08.826340 28603 rpcz_store.cc:276] 0503 14:12:06.227923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.264752 (+ 36829us) service_pool.cc:225] Handling call
0503 14:12:08.826046 (+2561294us) ts_tablet_manager.cc:1120] Deleting tablet 0b255d3e3dc64fb69c2ef6ed389db6a6
0503 14:12:08.826056 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.826160 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2550140}
I20240503 14:12:08.826612 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 182) took 3168 ms. Trace:
I20240503 14:12:08.826642 28603 tablet_replica.cc:330] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.826910 28603 raft_consensus.cc:2229] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.816047 28653 ts_tablet_manager.cc:1888] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:08.827142 28603 pending_rounds.cc:62] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:08.827164 28653 log.cc:1196] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/8ba629915ef44e0882beee2612e285c1
I20240503 14:12:08.827255 28603 pending_rounds.cc:68] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597672362061824 op_type: NO_OP noop_request { }
I20240503 14:12:08.827436 28603 raft_consensus.cc:2866] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:08.827617 28603 raft_consensus.cc:2258] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.827657 28653 ts_tablet_manager.cc:1909] T 8ba629915ef44e0882beee2612e285c1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.828593 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 232) took 2141 ms. Trace:
I20240503 14:12:08.828729 28653 rpcz_store.cc:276] 0503 14:12:06.744238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.744341 (+ 103us) service_pool.cc:225] Handling call
0503 14:12:08.780501 (+2036160us) ts_tablet_manager.cc:1120] Deleting tablet 8ba629915ef44e0882beee2612e285c1
0503 14:12:08.780512 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.812330 (+ 31818us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.815908 (+ 3578us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.828526 (+ 12618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035716}
I20240503 14:12:08.829007 28604 tablet_bootstrap.cc:492] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:08.829012 28603 ts_tablet_manager.cc:1875] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.829334 28604 ts_tablet_manager.cc:1367] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.000s
I20240503 14:12:08.829315 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 233) took 2083 ms. Trace:
I20240503 14:12:08.829429 28653 rpcz_store.cc:276] 0503 14:12:06.745736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.745811 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:08.829039 (+2083228us) ts_tablet_manager.cc:1120] Deleting tablet b25c852aa7aa4fbfafdacec43f5bc426
0503 14:12:08.829050 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.829226 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2082881}
I20240503 14:12:08.829917 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 234) took 2083 ms. Trace:
I20240503 14:12:08.830045 28653 rpcz_store.cc:276] 0503 14:12:06.746573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.746647 (+ 74us) service_pool.cc:225] Handling call
0503 14:12:08.829723 (+2083076us) ts_tablet_manager.cc:1120] Deleting tablet 75dafd98b05d4c63bbf30165a2ac8d1e
0503 14:12:08.829732 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.829822 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2082751}
W20240503 14:12:08.800071 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f50ad403bd354559b2522a76af1fa722 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50ad403bd354559b2522a76af1fa722 already in progress: creating tablet
I20240503 14:12:08.830494 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 250) took 1688 ms. Trace:
I20240503 14:12:08.830602 28653 rpcz_store.cc:276] 0503 14:12:07.142050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.142123 (+ 73us) service_pool.cc:225] Handling call
0503 14:12:08.830265 (+1688142us) ts_tablet_manager.cc:1120] Deleting tablet 1c5e3649c71f4d4da710c69f85a82c25
0503 14:12:08.830273 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.830401 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1670057}
I20240503 14:12:08.831080 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 253) took 1686 ms. Trace:
I20240503 14:12:08.831208 28653 rpcz_store.cc:276] 0503 14:12:07.144476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.144543 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:08.830830 (+1686287us) ts_tablet_manager.cc:1120] Deleting tablet c5297664a4d943c28adfb6ac4f061680
0503 14:12:08.830839 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.830977 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1670556}
I20240503 14:12:08.831657 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 251) took 1689 ms. Trace:
I20240503 14:12:08.831773 28653 rpcz_store.cc:276] 0503 14:12:07.142286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.142483 (+ 197us) service_pool.cc:225] Handling call
0503 14:12:08.831441 (+1688958us) ts_tablet_manager.cc:1120] Deleting tablet 50edc38cdbe74397aa4f4928d56d9fbc
0503 14:12:08.831450 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.831564 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1671118}
I20240503 14:12:08.824196 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:08.832322 28217 raft_consensus.cc:2454] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:08.832825 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 254) took 1665 ms. Trace:
I20240503 14:12:08.832940 28653 rpcz_store.cc:276] 0503 14:12:07.166848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.166958 (+ 110us) service_pool.cc:225] Handling call
0503 14:12:08.832024 (+1665066us) ts_tablet_manager.cc:1120] Deleting tablet e135b60660f448a7974520f2f87cf93a
0503 14:12:08.832033 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.832733 (+ 700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664687}
I20240503 14:12:08.833519 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 252) took 1691 ms. Trace:
I20240503 14:12:08.833633 28653 rpcz_store.cc:276] 0503 14:12:07.142399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.142595 (+ 196us) service_pool.cc:225] Handling call
0503 14:12:08.833301 (+1690706us) ts_tablet_manager.cc:1120] Deleting tablet 7abcb8658fc34c939aa74cf0bd981ae4
0503 14:12:08.833310 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.833420 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665111}
I20240503 14:12:08.834010 28653 tablet_replica.cc:330] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.834293 28653 raft_consensus.cc:2229] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.834547 28653 raft_consensus.cc:2258] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.835240 28653 ts_tablet_manager.cc:1875] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.836241 28604 raft_consensus.cc:348] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.837466 28604 raft_consensus.cc:374] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.837898 28604 raft_consensus.cc:729] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.837978 28603 ts_tablet_manager.cc:1888] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.838461 28604 consensus_queue.cc:260] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.841003 28603 log.cc:1196] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/9dbabe5beb9a42c69cb64e1b364d75a1
W20240503 14:12:08.830771 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 37dc7d96357743089372baf71654b1e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37dc7d96357743089372baf71654b1e3 already in progress: creating tablet
I20240503 14:12:08.841570 28603 ts_tablet_manager.cc:1909] T 9dbabe5beb9a42c69cb64e1b364d75a1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.841550 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ffbd3291c56b4c13813c80d6775afe6f (table default.Table_f99793320ab94248abb70c15ee57ca02 [id=cde896fd6d7345f998085c10bbdee418]) successfully deleted
I20240503 14:12:08.842075 28604 ts_tablet_manager.cc:1398] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:12:08.842643 29202 tablet.cc:1722] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:08.842990 28661 rpcz_store.cc:276] 0503 14:12:05.657775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.657844 (+ 69us) service_pool.cc:225] Handling call
0503 14:12:08.826410 (+3168566us) ts_tablet_manager.cc:1120] Deleting tablet 1044f34588d045ce898435b61a90503f
0503 14:12:08.826419 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.826532 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3167319}
I20240503 14:12:08.844000 28661 tablet_replica.cc:330] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:08.842928 28600 tablet_bootstrap.cc:492] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.844841 28661 raft_consensus.cc:2229] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:08.845435 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d2600031df4d77b9afe636a9b5ad40" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:08.845010 29200 tablet.cc:1722] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:08.845703 28357 raft_consensus.cc:3037] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.845561 28661 raft_consensus.cc:2258] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.846730 28600 tablet_bootstrap.cc:654] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.848527 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 225) took 2618 ms. Trace:
I20240503 14:12:08.848726 28603 rpcz_store.cc:276] 0503 14:12:06.230056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.275957 (+ 45901us) service_pool.cc:225] Handling call
0503 14:12:08.826541 (+2550584us) ts_tablet_manager.cc:1120] Deleting tablet 9dbabe5beb9a42c69cb64e1b364d75a1
0503 14:12:08.826550 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.832947 (+ 6397us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.837794 (+ 4847us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.848447 (+ 10653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2509206}
I20240503 14:12:08.848917 28357 raft_consensus.cc:2454] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:08.849216 28603 tablet_replica.cc:330] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.849344 28661 ts_tablet_manager.cc:1875] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:08.851460 28427 leader_election.cc:341] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:08.852294 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 already in progress: creating tablet
W20240503 14:12:08.852756 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5775006e9f8f4612aa70dfb316de9ea1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5775006e9f8f4612aa70dfb316de9ea1 already in progress: creating tablet
I20240503 14:12:08.853132 28427 leader_election.cc:302] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:08.855660 28603 raft_consensus.cc:2229] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:08.856261 28603 raft_consensus.cc:2258] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.860005 28653 ts_tablet_manager.cc:1888] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.860854 28603 ts_tablet_manager.cc:1875] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:08.861317 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 06fb79fcf30d485798c64c815e372dc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06fb79fcf30d485798c64c815e372dc1 already in progress: creating tablet
I20240503 14:12:08.861626 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet eee989ff083041dc8dcd2cd1b220780d (table default.Table_706c4da08fa84bbaa207fc71eb482682 [id=36f9af99d9d44c32876688fc8cbc4f8b]) successfully deleted
W20240503 14:12:08.861897 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 65cdd090bf0f493f8b92341369378532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65cdd090bf0f493f8b92341369378532 already in progress: creating tablet
I20240503 14:12:08.862167 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]) successfully deleted
W20240503 14:12:08.862411 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b25c852aa7aa4fbfafdacec43f5bc426 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25c852aa7aa4fbfafdacec43f5bc426 already in progress: creating tablet
W20240503 14:12:08.862671 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 75dafd98b05d4c63bbf30165a2ac8d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75dafd98b05d4c63bbf30165a2ac8d1e already in progress: creating tablet
W20240503 14:12:08.862922 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 1c5e3649c71f4d4da710c69f85a82c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c5e3649c71f4d4da710c69f85a82c25 already in progress: creating tablet
I20240503 14:12:08.842999 28604 tablet_bootstrap.cc:492] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:08.865005 28661 ts_tablet_manager.cc:1888] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.865335 28653 log.cc:1196] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e355aafb2f5b4337bedaac6fe5539dc2
I20240503 14:12:08.866894 28653 ts_tablet_manager.cc:1909] T e355aafb2f5b4337bedaac6fe5539dc2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.865522 28604 tablet_bootstrap.cc:654] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.866458 29180 raft_consensus.cc:2781] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.868050 29180 raft_consensus.cc:686] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:08.868582 29180 consensus_queue.cc:237] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:08.876281 28600 tablet_bootstrap.cc:492] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.876652 28600 ts_tablet_manager.cc:1367] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.002s
I20240503 14:12:08.878002 28614 tablet_bootstrap.cc:654] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.878228 28600 raft_consensus.cc:348] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.878652 28600 raft_consensus.cc:374] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.879153 28600 raft_consensus.cc:729] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.879354 28661 log.cc:1196] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/1693dd36a3d84d88a49690c537590083
I20240503 14:12:08.879621 28600 consensus_queue.cc:260] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.880465 28600 ts_tablet_manager.cc:1398] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:08.881793 28600 tablet_bootstrap.cc:492] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.881333 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 255) took 1712 ms. Trace:
I20240503 14:12:08.882462 28653 rpcz_store.cc:276] 0503 14:12:07.168961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.169112 (+ 151us) service_pool.cc:225] Handling call
0503 14:12:08.833839 (+1664727us) ts_tablet_manager.cc:1120] Deleting tablet e355aafb2f5b4337bedaac6fe5539dc2
0503 14:12:08.833847 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.839096 (+ 5249us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.859818 (+ 20722us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.881246 (+ 21428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1662352}
I20240503 14:12:08.883308 28603 ts_tablet_manager.cc:1888] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:08.883625 28603 log.cc:1196] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/eee989ff083041dc8dcd2cd1b220780d
I20240503 14:12:08.883985 28653 tablet_replica.cc:330] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.884323 28603 ts_tablet_manager.cc:1909] T eee989ff083041dc8dcd2cd1b220780d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.885365 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 226) took 2649 ms. Trace:
I20240503 14:12:08.885589 28603 rpcz_store.cc:276] 0503 14:12:06.235963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.317387 (+ 81424us) service_pool.cc:225] Handling call
0503 14:12:08.849063 (+2531676us) ts_tablet_manager.cc:1120] Deleting tablet eee989ff083041dc8dcd2cd1b220780d
0503 14:12:08.849074 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.864800 (+ 15726us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.883141 (+ 18341us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.885281 (+ 2140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2514857}
I20240503 14:12:08.885927 28600 tablet_bootstrap.cc:654] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.886452 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 228) took 2645 ms. Trace:
I20240503 14:12:08.886654 28603 rpcz_store.cc:276] 0503 14:12:06.241040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.406650 (+165610us) service_pool.cc:225] Handling call
0503 14:12:08.886197 (+2479547us) ts_tablet_manager.cc:1120] Deleting tablet 5775006e9f8f4612aa70dfb316de9ea1
0503 14:12:08.886210 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.886358 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2479253}
I20240503 14:12:08.886844 28427 leader_election.cc:302] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:08.887467 28653 raft_consensus.cc:2229] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.887742 28653 raft_consensus.cc:2258] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.887730 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 239) took 2349 ms. Trace:
I20240503 14:12:08.887964 28603 rpcz_store.cc:276] 0503 14:12:06.538164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.538620 (+ 456us) service_pool.cc:225] Handling call
0503 14:12:08.887314 (+2348694us) ts_tablet_manager.cc:1120] Deleting tablet 06fb79fcf30d485798c64c815e372dc1
0503 14:12:08.887325 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.887548 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208343}
I20240503 14:12:08.888506 28653 ts_tablet_manager.cc:1875] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.888818 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 242) took 2218 ms. Trace:
I20240503 14:12:08.888947 28661 ts_tablet_manager.cc:1909] T 1693dd36a3d84d88a49690c537590083 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:08.888995 28603 rpcz_store.cc:276] 0503 14:12:06.670399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.670972 (+ 573us) service_pool.cc:225] Handling call
0503 14:12:08.888515 (+2217543us) ts_tablet_manager.cc:1120] Deleting tablet b25c852aa7aa4fbfafdacec43f5bc426
0503 14:12:08.888527 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.888734 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208550}
I20240503 14:12:08.889675 28603 tablet_replica.cc:330] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.889973 28603 raft_consensus.cc:2229] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:08.889999 29182 raft_consensus.cc:2781] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.890146 29182 raft_consensus.cc:482] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.890306 29182 raft_consensus.cc:3037] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.890656 28603 raft_consensus.cc:2258] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.891448 28603 ts_tablet_manager.cc:1875] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.893860 28604 tablet_bootstrap.cc:492] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:08.894125 28604 ts_tablet_manager.cc:1367] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.020s
I20240503 14:12:08.894603 29182 raft_consensus.cc:504] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.895227 28604 raft_consensus.cc:348] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.895478 28604 raft_consensus.cc:374] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.895633 28604 raft_consensus.cc:729] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.895897 28604 consensus_queue.cc:260] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.896582 28604 ts_tablet_manager.cc:1398] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:08.896975 28604 tablet_bootstrap.cc:492] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:08.899011 28604 tablet_bootstrap.cc:654] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.901655 28600 tablet_bootstrap.cc:492] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.901947 28600 ts_tablet_manager.cc:1367] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20240503 14:12:08.903123 28600 raft_consensus.cc:348] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.903402 28600 raft_consensus.cc:374] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.903491 28600 raft_consensus.cc:729] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.903823 28600 consensus_queue.cc:260] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.904214 28653 ts_tablet_manager.cc:1888] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:08.904202 28427 leader_election.cc:341] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a8d2600031df4d77b9afe636a9b5ad40
I20240503 14:12:08.905092 28600 ts_tablet_manager.cc:1398] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:08.905663 28653 log.cc:1196] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/351d5bfb12c34c5f9de90e2df1b36fc1
I20240503 14:12:08.906638 28653 ts_tablet_manager.cc:1909] T 351d5bfb12c34c5f9de90e2df1b36fc1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.912743 28603 ts_tablet_manager.cc:1888] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:08.895489 29182 leader_election.cc:288] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:08.913154 28603 log.cc:1196] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/8ba629915ef44e0882beee2612e285c1
I20240503 14:12:08.913789 28603 ts_tablet_manager.cc:1909] T 8ba629915ef44e0882beee2612e285c1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.905979 28600 tablet_bootstrap.cc:492] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.914816 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 241) took 2244 ms. Trace:
I20240503 14:12:08.915127 28603 rpcz_store.cc:276] 0503 14:12:06.670291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.670858 (+ 567us) service_pool.cc:225] Handling call
0503 14:12:08.889546 (+2218688us) ts_tablet_manager.cc:1120] Deleting tablet 8ba629915ef44e0882beee2612e285c1
0503 14:12:08.889557 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.899386 (+ 9829us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.912534 (+ 13148us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.914769 (+ 2235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2204222}
I20240503 14:12:08.917008 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 240) took 2246 ms. Trace:
I20240503 14:12:08.917270 28603 rpcz_store.cc:276] 0503 14:12:06.670114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.670825 (+ 711us) service_pool.cc:225] Handling call
0503 14:12:08.916704 (+2245879us) ts_tablet_manager.cc:1120] Deleting tablet 65cdd090bf0f493f8b92341369378532
0503 14:12:08.916716 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.916943 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230446}
I20240503 14:12:08.917640 28600 tablet_bootstrap.cc:654] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:08.918051 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c5297664a4d943c28adfb6ac4f061680 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5297664a4d943c28adfb6ac4f061680 already in progress: creating tablet
W20240503 14:12:08.918437 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 50edc38cdbe74397aa4f4928d56d9fbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50edc38cdbe74397aa4f4928d56d9fbc already in progress: creating tablet
W20240503 14:12:08.918761 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e135b60660f448a7974520f2f87cf93a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e135b60660f448a7974520f2f87cf93a already in progress: creating tablet
W20240503 14:12:08.919018 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7abcb8658fc34c939aa74cf0bd981ae4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7abcb8658fc34c939aa74cf0bd981ae4 already in progress: creating tablet
I20240503 14:12:08.919283 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e355aafb2f5b4337bedaac6fe5539dc2 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
I20240503 14:12:08.919548 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 243) took 2248 ms. Trace:
I20240503 14:12:08.919773 28603 rpcz_store.cc:276] 0503 14:12:06.670515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.679023 (+ 8508us) service_pool.cc:225] Handling call
0503 14:12:08.918805 (+2239782us) ts_tablet_manager.cc:1120] Deleting tablet 75dafd98b05d4c63bbf30165a2ac8d1e
0503 14:12:08.918818 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.919072 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2217136}
I20240503 14:12:08.919998 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 261) took 1557 ms. Trace:
I20240503 14:12:08.920183 28653 rpcz_store.cc:276] 0503 14:12:07.363005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.363256 (+ 251us) service_pool.cc:225] Handling call
0503 14:12:08.883846 (+1520590us) ts_tablet_manager.cc:1120] Deleting tablet 351d5bfb12c34c5f9de90e2df1b36fc1
0503 14:12:08.883861 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.894799 (+ 10938us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.904014 (+ 9215us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.919917 (+ 15903us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1520085}
I20240503 14:12:08.921478 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 263) took 1554 ms. Trace:
I20240503 14:12:08.921633 28653 rpcz_store.cc:276] 0503 14:12:07.366866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.366931 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:08.921177 (+1554246us) ts_tablet_manager.cc:1120] Deleting tablet 1d77014c66624f958b38f7cc83644891
0503 14:12:08.921192 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.921372 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528890}
I20240503 14:12:08.922438 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 259) took 1742 ms. Trace:
I20240503 14:12:08.922629 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 262) took 1558 ms. Trace:
I20240503 14:12:08.922750 28653 rpcz_store.cc:276] 0503 14:12:07.364313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.373067 (+ 8754us) service_pool.cc:225] Handling call
0503 14:12:08.922440 (+1549373us) ts_tablet_manager.cc:1120] Deleting tablet 74df7781cc6a4dd6b6fd78a728ffd71b
0503 14:12:08.922452 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.922569 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526218}
I20240503 14:12:08.923377 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 264) took 1373 ms. Trace:
I20240503 14:12:08.923516 28653 rpcz_store.cc:276] 0503 14:12:07.549565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.549668 (+ 103us) service_pool.cc:225] Handling call
0503 14:12:08.923177 (+1373509us) ts_tablet_manager.cc:1120] Deleting tablet 2404807f8500437ababca53f6388b636
0503 14:12:08.923188 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.923302 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373171}
I20240503 14:12:08.922854 28603 rpcz_store.cc:276] 0503 14:12:07.180077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.180190 (+ 113us) service_pool.cc:225] Handling call
0503 14:12:08.922223 (+1742033us) ts_tablet_manager.cc:1120] Deleting tablet 1c5e3649c71f4d4da710c69f85a82c25
0503 14:12:08.922236 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.922377 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1721133}
I20240503 14:12:08.924137 28653 tablet_replica.cc:330] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:08.924394 28653 raft_consensus.cc:2229] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.924721 28653 raft_consensus.cc:2258] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.925058 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 261) took 1681 ms. Trace:
I20240503 14:12:08.925102 28600 tablet_bootstrap.cc:492] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:08.925297 28600 ts_tablet_manager.cc:1367] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20240503 14:12:08.925688 28653 ts_tablet_manager.cc:1875] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.925913 28603 rpcz_store.cc:276] 0503 14:12:07.244066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.244155 (+ 89us) service_pool.cc:225] Handling call
0503 14:12:08.924454 (+1680299us) ts_tablet_manager.cc:1120] Deleting tablet 7abcb8658fc34c939aa74cf0bd981ae4
0503 14:12:08.924463 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.925002 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672627}
I20240503 14:12:08.926402 28600 raft_consensus.cc:348] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.926692 28600 raft_consensus.cc:374] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.926853 28600 raft_consensus.cc:729] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.927165 28600 consensus_queue.cc:260] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:08.927805 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 260) took 1729 ms. Trace:
I20240503 14:12:08.927847 28600 ts_tablet_manager.cc:1398] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:08.928017 28603 rpcz_store.cc:276] 0503 14:12:07.199541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.199751 (+ 210us) service_pool.cc:225] Handling call
0503 14:12:08.926841 (+1727090us) ts_tablet_manager.cc:1120] Deleting tablet 50edc38cdbe74397aa4f4928d56d9fbc
0503 14:12:08.926854 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.927746 (+ 892us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1674112}
I20240503 14:12:08.928252 28600 tablet_bootstrap.cc:492] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:08.930732 28600 tablet_bootstrap.cc:654] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.931510 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 262) took 1685 ms. Trace:
I20240503 14:12:08.931794 28603 rpcz_store.cc:276] 0503 14:12:07.246444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.246529 (+ 85us) service_pool.cc:225] Handling call
0503 14:12:08.931258 (+1684729us) ts_tablet_manager.cc:1120] Deleting tablet c5297664a4d943c28adfb6ac4f061680
0503 14:12:08.931270 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.931442 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1626979}
I20240503 14:12:08.932233 28604 tablet_bootstrap.cc:492] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:08.932474 28604 ts_tablet_manager.cc:1367] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.013s
I20240503 14:12:08.933144 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 263) took 1665 ms. Trace:
I20240503 14:12:08.933485 28604 raft_consensus.cc:348] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.933800 28604 raft_consensus.cc:374] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.933923 28604 raft_consensus.cc:729] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.933674 28614 tablet_bootstrap.cc:492] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:08.934259 28604 consensus_queue.cc:260] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.934614 28614 ts_tablet_manager.cc:1367] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.116s user 0.001s sys 0.004s
I20240503 14:12:08.934134 28603 rpcz_store.cc:276] 0503 14:12:07.267809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.267938 (+ 129us) service_pool.cc:225] Handling call
0503 14:12:08.932928 (+1664990us) ts_tablet_manager.cc:1120] Deleting tablet e135b60660f448a7974520f2f87cf93a
0503 14:12:08.932939 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.933088 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614268}
I20240503 14:12:08.934907 28604 ts_tablet_manager.cc:1398] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:08.935324 28604 tablet_bootstrap.cc:492] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:08.936319 28614 raft_consensus.cc:348] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.937925 28614 raft_consensus.cc:374] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.938160 28614 raft_consensus.cc:729] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.939829 28614 consensus_queue.cc:260] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.924857 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 183) took 3263 ms. Trace:
I20240503 14:12:08.938180 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:08.940171 28661 rpcz_store.cc:276] 0503 14:12:05.661143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.668478 (+ 7335us) service_pool.cc:225] Handling call
0503 14:12:08.843884 (+3175406us) ts_tablet_manager.cc:1120] Deleting tablet 1693dd36a3d84d88a49690c537590083
0503 14:12:08.843895 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.860473 (+ 16578us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.864825 (+ 4352us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.924777 (+ 59952us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3175070}
I20240503 14:12:08.941066 28614 ts_tablet_manager.cc:1398] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:08.946403 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:08.946847 28212 raft_consensus.cc:3037] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.950037 28604 tablet_bootstrap.cc:654] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:08.950382 28427 leader_election.cc:341] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:08.951046 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 184) took 3255 ms. Trace:
I20240503 14:12:08.951413 28212 raft_consensus.cc:2454] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:08.938939 29204 tablet.cc:1722] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:08.936599 28603 tablet_replica.cc:330] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:08.953958 28603 raft_consensus.cc:2229] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.954282 28603 raft_consensus.cc:2258] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.955036 28603 ts_tablet_manager.cc:1875] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.958887 28653 ts_tablet_manager.cc:1888] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.959210 28653 log.cc:1196] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6dab0296a0894a63afc16ed466c591d5
I20240503 14:12:08.959801 28653 ts_tablet_manager.cc:1909] T 6dab0296a0894a63afc16ed466c591d5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:08.960531 28614 tablet_bootstrap.cc:492] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:08.960907 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 265) took 1410 ms. Trace:
I20240503 14:12:08.961117 28653 rpcz_store.cc:276] 0503 14:12:07.550562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.550626 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:08.923994 (+1373368us) ts_tablet_manager.cc:1120] Deleting tablet 6dab0296a0894a63afc16ed466c591d5
0503 14:12:08.924001 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.929887 (+ 5886us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.935022 (+ 5135us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.960856 (+ 25834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373013}
I20240503 14:12:08.962046 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 266) took 1410 ms. Trace:
I20240503 14:12:08.962877 28614 tablet_bootstrap.cc:654] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:08.963586 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 63e977f7837e49fb93ef0fd18b619792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e977f7837e49fb93ef0fd18b619792 already in progress: creating tablet
W20240503 14:12:08.964051 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7c6f41bb9c5d40b0913264864ecc3528 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6f41bb9c5d40b0913264864ecc3528 already in progress: creating tablet
I20240503 14:12:08.969594 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 8bfc56c4591c437da5b9686b726686b2 (table default.Table_955617b294c94c299632de62b5973057 [id=3a24797cc53b439da44cc77025fa9f92]) successfully deleted
I20240503 14:12:08.970105 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet fde0e59d911d4c1eb5125b1f869b396c (table default.Table_666a0b3b2afb4a9fb91fb970e6c7a173 [id=a66a34594c864f0c81ba2da0a1449aee]) successfully deleted
I20240503 14:12:08.971493 28653 rpcz_store.cc:276] 0503 14:12:07.551560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.551617 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:08.961808 (+1410191us) ts_tablet_manager.cc:1120] Deleting tablet e0a92fc3833e49aaab8c5415de9bb685
0503 14:12:08.961831 (+ 23us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.961981 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409947}
I20240503 14:12:08.972321 28427 leader_election.cc:302] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:08.974570 28603 ts_tablet_manager.cc:1888] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:08.972226 28661 rpcz_store.cc:276] 0503 14:12:05.685655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.685831 (+ 176us) service_pool.cc:225] Handling call
0503 14:12:08.940828 (+3254997us) ts_tablet_manager.cc:1120] Deleting tablet 129717ef137c41c5a08d401c64164187
0503 14:12:08.940841 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.941010 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3254698}
I20240503 14:12:08.975191 28661 tablet_replica.cc:330] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:08.975579 28661 raft_consensus.cc:2229] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:08.975896 28661 raft_consensus.cc:2258] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:08.976598 29182 raft_consensus.cc:2781] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.977022 29182 raft_consensus.cc:686] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:08.977118 28661 ts_tablet_manager.cc:1875] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:08.977430 29182 consensus_queue.cc:237] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.980363 28603 log.cc:1196] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e355aafb2f5b4337bedaac6fe5539dc2
I20240503 14:12:08.980861 28603 ts_tablet_manager.cc:1909] T e355aafb2f5b4337bedaac6fe5539dc2 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:08.981786 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 264) took 1675 ms. Trace:
I20240503 14:12:08.981922 28603 rpcz_store.cc:276] 0503 14:12:07.306698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.306761 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:08.935278 (+1628517us) ts_tablet_manager.cc:1120] Deleting tablet e355aafb2f5b4337bedaac6fe5539dc2
0503 14:12:08.935290 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.959108 (+ 23818us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.974418 (+ 15310us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:08.981718 (+ 7300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599022}
I20240503 14:12:08.974916 28614 tablet_bootstrap.cc:492] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
W20240503 14:12:08.971115 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1044f34588d045ce898435b61a90503f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1044f34588d045ce898435b61a90503f already in progress: creating tablet
I20240503 14:12:08.982545 28614 ts_tablet_manager.cc:1367] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s
I20240503 14:12:08.982978 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 1693dd36a3d84d88a49690c537590083 (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
W20240503 14:12:08.983402 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 129717ef137c41c5a08d401c64164187 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 129717ef137c41c5a08d401c64164187 already in progress: creating tablet
I20240503 14:12:08.983585 28614 raft_consensus.cc:348] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.983868 28614 raft_consensus.cc:374] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.984014 28614 raft_consensus.cc:729] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.984309 28614 consensus_queue.cc:260] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.984910 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 4775ff1029e6449c83219030e3809984 (table default.Table_2ad02256d653464e9528725b5e37c72c [id=36785e97f5c94a15bf717d499a33f558]) successfully deleted
I20240503 14:12:08.985047 28614 ts_tablet_manager.cc:1398] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:08.985270 28604 tablet_bootstrap.cc:492] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:08.985504 28604 ts_tablet_manager.cc:1367] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.004s
I20240503 14:12:08.986855 28614 tablet_bootstrap.cc:492] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:08.989228 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_85ee5bde7ad044389efb107eefd39fd3 [id=11af424ea24549bc83f6203448ae619c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5b90e9563b4c4cef8ee7964eaa96e285 consensus state has no leader
I20240503 14:12:08.989362 28614 tablet_bootstrap.cc:654] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:08.989995 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 270) took 1482 ms. Trace:
I20240503 14:12:08.990160 28603 rpcz_store.cc:276] 0503 14:12:07.507522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.507580 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:08.989728 (+1482148us) ts_tablet_manager.cc:1120] Deleting tablet 351d5bfb12c34c5f9de90e2df1b36fc1
0503 14:12:08.989741 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.989891 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479681}
I20240503 14:12:08.993916 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 271) took 1481 ms. Trace:
I20240503 14:12:08.994186 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 267) took 1381 ms. Trace:
I20240503 14:12:08.994115 28603 rpcz_store.cc:276] 0503 14:12:07.512900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.512957 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:08.993642 (+1480685us) ts_tablet_manager.cc:1120] Deleting tablet 74df7781cc6a4dd6b6fd78a728ffd71b
0503 14:12:08.993656 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.993818 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474760}
I20240503 14:12:08.994479 28653 rpcz_store.cc:276] 0503 14:12:07.612819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.612889 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:08.993817 (+1380928us) ts_tablet_manager.cc:1120] Deleting tablet 6371e68ee85a47cdb0e8e21f7124d94a
0503 14:12:08.993829 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.994036 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380607}
I20240503 14:12:08.995242 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 277) took 1155 ms. Trace:
I20240503 14:12:08.995513 28653 rpcz_store.cc:276] 0503 14:12:07.839665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.839740 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:08.995036 (+1155296us) ts_tablet_manager.cc:1120] Deleting tablet 140de4d395d54b7696062ed96b8aa89c
0503 14:12:08.995048 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.995178 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153924}
I20240503 14:12:08.995817 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 272) took 1481 ms. Trace:
I20240503 14:12:08.995949 28603 rpcz_store.cc:276] 0503 14:12:07.514496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.516892 (+ 2396us) service_pool.cc:225] Handling call
0503 14:12:08.995591 (+1478699us) ts_tablet_manager.cc:1120] Deleting tablet 1d77014c66624f958b38f7cc83644891
0503 14:12:08.995601 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.995725 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1476347}
I20240503 14:12:08.991693 28604 raft_consensus.cc:348] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.996559 28604 raft_consensus.cc:374] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:08.996635 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 273) took 1392 ms. Trace:
I20240503 14:12:08.996764 28603 rpcz_store.cc:276] 0503 14:12:07.604383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.604437 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:08.996419 (+1391982us) ts_tablet_manager.cc:1120] Deleting tablet 2404807f8500437ababca53f6388b636
0503 14:12:08.996430 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.996546 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379509}
I20240503 14:12:08.997431 28604 raft_consensus.cc:729] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:08.997917 28604 consensus_queue.cc:260] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:08.998839 28604 ts_tablet_manager.cc:1398] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:12:08.999271 28604 tablet_bootstrap.cc:492] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.000339 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 274) took 1382 ms. Trace:
I20240503 14:12:09.000556 28603 rpcz_store.cc:276] 0503 14:12:07.617499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.617623 (+ 124us) service_pool.cc:225] Handling call
0503 14:12:08.998889 (+1381266us) ts_tablet_manager.cc:1120] Deleting tablet 6dab0296a0894a63afc16ed466c591d5
0503 14:12:08.998902 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.000239 (+ 1337us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379950}
W20240503 14:12:09.000998 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 37dc7d96357743089372baf71654b1e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37dc7d96357743089372baf71654b1e3 already in progress: creating tablet
W20240503 14:12:09.001446 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ffbd3291c56b4c13813c80d6775afe6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffbd3291c56b4c13813c80d6775afe6f already in progress: creating tablet
I20240503 14:12:09.001650 28604 tablet_bootstrap.cc:654] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.001866 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 275) took 1383 ms. Trace:
I20240503 14:12:08.997488 28600 tablet_bootstrap.cc:492] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.002370 28600 ts_tablet_manager.cc:1367] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.074s user 0.023s sys 0.036s
I20240503 14:12:09.003436 28600 raft_consensus.cc:348] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.003796 28600 raft_consensus.cc:374] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.002282 28603 rpcz_store.cc:276] 0503 14:12:07.618551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.618662 (+ 111us) service_pool.cc:225] Handling call
0503 14:12:09.001103 (+1382441us) ts_tablet_manager.cc:1120] Deleting tablet e0a92fc3833e49aaab8c5415de9bb685
0503 14:12:09.001621 (+ 518us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.001769 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1370192}
I20240503 14:12:09.003912 28600 raft_consensus.cc:729] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.004374 28600 consensus_queue.cc:260] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.006162 28600 ts_tablet_manager.cc:1398] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:09.009244 28661 ts_tablet_manager.cc:1888] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.009433 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 278) took 1158 ms. Trace:
I20240503 14:12:09.009620 28653 rpcz_store.cc:276] 0503 14:12:07.851094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.851159 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:09.005428 (+1154269us) ts_tablet_manager.cc:1120] Deleting tablet 962df44408484e3f8d7b7138f27c1a66
0503 14:12:09.009199 (+ 3771us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.009366 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153934}
W20240503 14:12:09.010241 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 already in progress: creating tablet
I20240503 14:12:09.010641 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 9dbabe5beb9a42c69cb64e1b364d75a1 (table default.Table_6d7feb39cce84e66b75914a6bcbcfcc9 [id=56b86fbb2fff40cfa3729f88092d57ca]) successfully deleted
I20240503 14:12:09.010684 28661 log.cc:1196] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/f51834695b8b4a729a088da81eee2e47
I20240503 14:12:09.010910 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet eee989ff083041dc8dcd2cd1b220780d (table default.Table_706c4da08fa84bbaa207fc71eb482682 [id=36f9af99d9d44c32876688fc8cbc4f8b]) successfully deleted
W20240503 14:12:09.011207 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5775006e9f8f4612aa70dfb316de9ea1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5775006e9f8f4612aa70dfb316de9ea1 already in progress: creating tablet
W20240503 14:12:09.011461 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 06fb79fcf30d485798c64c815e372dc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06fb79fcf30d485798c64c815e372dc1 already in progress: creating tablet
I20240503 14:12:09.011495 28661 ts_tablet_manager.cc:1909] T f51834695b8b4a729a088da81eee2e47 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.012301 28600 tablet_bootstrap.cc:492] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.012723 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 185) took 3094 ms. Trace:
I20240503 14:12:09.012957 28661 rpcz_store.cc:276] 0503 14:12:05.918110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.918344 (+ 234us) service_pool.cc:225] Handling call
0503 14:12:08.975023 (+3056679us) ts_tablet_manager.cc:1120] Deleting tablet f51834695b8b4a729a088da81eee2e47
0503 14:12:08.975038 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:08.986888 (+ 11850us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.009088 (+ 22200us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.012621 (+ 3533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3053947}
I20240503 14:12:09.013870 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 188) took 3092 ms. Trace:
I20240503 14:12:09.014117 28661 rpcz_store.cc:276] 0503 14:12:05.921048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.921280 (+ 232us) service_pool.cc:225] Handling call
0503 14:12:09.013630 (+3092350us) ts_tablet_manager.cc:1120] Deleting tablet 4775ff1029e6449c83219030e3809984
0503 14:12:09.013644 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.013769 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3089899}
I20240503 14:12:09.014532 28600 tablet_bootstrap.cc:654] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.014930 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 187) took 3094 ms. Trace:
I20240503 14:12:09.015131 28661 rpcz_store.cc:276] 0503 14:12:05.920874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.921248 (+ 374us) service_pool.cc:225] Handling call
0503 14:12:09.014705 (+3093457us) ts_tablet_manager.cc:1120] Deleting tablet 7fe5d280a1654d80a2176c1def459e2e
0503 14:12:09.014717 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.014829 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3090152}
I20240503 14:12:09.015556 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 276) took 1373 ms. Trace:
I20240503 14:12:09.015708 28603 rpcz_store.cc:276] 0503 14:12:07.631074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.632688 (+ 1614us) service_pool.cc:225] Handling call
0503 14:12:09.004441 (+1371753us) ts_tablet_manager.cc:1120] Deleting tablet 6371e68ee85a47cdb0e8e21f7124d94a
0503 14:12:09.004453 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.004603 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363296}
I20240503 14:12:09.016999 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 286) took 1175 ms. Trace:
I20240503 14:12:09.018932 28653 tablet_replica.cc:330] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.019243 28653 raft_consensus.cc:2229] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.019502 28653 raft_consensus.cc:2258] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.019902 28614 tablet_bootstrap.cc:492] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.020227 28614 ts_tablet_manager.cc:1367] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240503 14:12:09.021593 28653 ts_tablet_manager.cc:1875] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.022089 28604 tablet_bootstrap.cc:492] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.022027 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 186) took 3101 ms. Trace:
I20240503 14:12:09.022449 28604 ts_tablet_manager.cc:1367] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.006s
I20240503 14:12:09.023166 28614 raft_consensus.cc:348] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.023476 28614 raft_consensus.cc:374] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.023597 28614 raft_consensus.cc:729] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.023519 28604 raft_consensus.cc:348] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.023768 28604 raft_consensus.cc:374] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.023854 28604 raft_consensus.cc:729] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.023947 28614 consensus_queue.cc:260] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.023433 28661 rpcz_store.cc:276] 0503 14:12:05.920218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.920324 (+ 106us) service_pool.cc:225] Handling call
0503 14:12:09.016147 (+3095823us) ts_tablet_manager.cc:1120] Deleting tablet 9dbabe5beb9a42c69cb64e1b364d75a1
0503 14:12:09.016158 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.021834 (+ 5676us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3089717}
I20240503 14:12:09.024128 28604 consensus_queue.cc:260] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.024686 28614 ts_tablet_manager.cc:1398] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:09.024696 28604 ts_tablet_manager.cc:1398] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:09.020120 28603 rpcz_store.cc:276] 0503 14:12:07.841832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.841908 (+ 76us) service_pool.cc:225] Handling call
0503 14:12:09.016731 (+1174823us) ts_tablet_manager.cc:1120] Deleting tablet 140de4d395d54b7696062ed96b8aa89c
0503 14:12:09.016741 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.016902 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174506}
I20240503 14:12:09.025015 28604 tablet_bootstrap.cc:492] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.025269 28614 tablet_bootstrap.cc:492] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.026377 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 189) took 3105 ms. Trace:
I20240503 14:12:09.026504 28661 rpcz_store.cc:276] 0503 14:12:05.921193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.921553 (+ 360us) service_pool.cc:225] Handling call
0503 14:12:09.026154 (+3104601us) ts_tablet_manager.cc:1120] Deleting tablet bacf2d1091ee458c961c691093ea65a9
0503 14:12:09.026166 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.026282 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3098463}
I20240503 14:12:09.026774 28604 tablet_bootstrap.cc:654] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.026916 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 287) took 1176 ms. Trace:
I20240503 14:12:09.027038 28603 rpcz_store.cc:276] 0503 14:12:07.850805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.850865 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:09.024918 (+1174053us) ts_tablet_manager.cc:1120] Deleting tablet 962df44408484e3f8d7b7138f27c1a66
0503 14:12:09.026669 (+ 1751us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.026830 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171279}
I20240503 14:12:09.027305 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 190) took 3100 ms. Trace:
I20240503 14:12:09.027422 28661 rpcz_store.cc:276] 0503 14:12:05.927276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.927440 (+ 164us) service_pool.cc:225] Handling call
0503 14:12:09.027099 (+3099659us) ts_tablet_manager.cc:1120] Deleting tablet ea3e9c030a424740bb2fb641328adfe1
0503 14:12:09.027110 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.027208 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3065050}
I20240503 14:12:09.027686 28603 tablet_replica.cc:330] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:09.027913 28603 raft_consensus.cc:2229] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:09.028318 28603 pending_rounds.cc:62] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:09.028424 28603 pending_rounds.cc:68] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597682372255744 op_type: NO_OP noop_request { }
I20240503 14:12:09.028652 28603 raft_consensus.cc:2866] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:09.028792 28603 raft_consensus.cc:2258] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.028110 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 192) took 3099 ms. Trace:
I20240503 14:12:09.029573 28603 ts_tablet_manager.cc:1875] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.029606 28661 rpcz_store.cc:276] 0503 14:12:05.928200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.960356 (+ 32156us) service_pool.cc:225] Handling call
0503 14:12:09.027828 (+3067472us) ts_tablet_manager.cc:1120] Deleting tablet 47ba571a81eb4240a4a4a4a08cb879c5
0503 14:12:09.027836 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.027995 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3065556}
I20240503 14:12:09.030388 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 191) took 3102 ms. Trace:
I20240503 14:12:09.030581 28661 rpcz_store.cc:276] 0503 14:12:05.927961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:05.961371 (+ 33410us) service_pool.cc:225] Handling call
0503 14:12:09.030164 (+3068793us) ts_tablet_manager.cc:1120] Deleting tablet eb79edba14b646ef9be239b05a291d71
0503 14:12:09.030175 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.030306 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3067588}
W20240503 14:12:09.011792 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b25c852aa7aa4fbfafdacec43f5bc426 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25c852aa7aa4fbfafdacec43f5bc426 already in progress: creating tablet
I20240503 14:12:09.031105 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]) successfully deleted
I20240503 14:12:09.031574 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 207) took 3030 ms. Trace:
I20240503 14:12:09.031754 28661 rpcz_store.cc:276] 0503 14:12:06.000752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.001935 (+ 1183us) service_pool.cc:225] Handling call
0503 14:12:09.031089 (+3029154us) ts_tablet_manager.cc:1120] Deleting tablet f50ad403bd354559b2522a76af1fa722
0503 14:12:09.031099 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.031277 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3028933}
I20240503 14:12:09.032416 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 208) took 3028 ms. Trace:
W20240503 14:12:09.031476 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 65cdd090bf0f493f8b92341369378532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65cdd090bf0f493f8b92341369378532 already in progress: creating tablet
I20240503 14:12:09.032840 29113 raft_consensus.cc:482] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.033006 28661 rpcz_store.cc:276] 0503 14:12:06.003440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.003503 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:09.032222 (+3028719us) ts_tablet_manager.cc:1120] Deleting tablet 37dc7d96357743089372baf71654b1e3
0503 14:12:09.032231 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.032340 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3027480}
I20240503 14:12:09.033053 29113 raft_consensus.cc:504] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.033955 29113 leader_election.cc:288] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.034965 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75dafd98b05d4c63bbf30165a2ac8d1e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.034973 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75dafd98b05d4c63bbf30165a2ac8d1e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:09.035547 28150 leader_election.cc:341] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:09.035871 28150 leader_election.cc:341] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.035998 28150 leader_election.cc:302] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.036048 28661 tablet_replica.cc:330] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.036338 28661 raft_consensus.cc:2229] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.036540 29113 raft_consensus.cc:2726] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.037037 28614 tablet_bootstrap.cc:654] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.037940 28661 raft_consensus.cc:2258] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:09.039552 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 75dafd98b05d4c63bbf30165a2ac8d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75dafd98b05d4c63bbf30165a2ac8d1e already in progress: creating tablet
W20240503 14:12:09.040035 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1c5e3649c71f4d4da710c69f85a82c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c5e3649c71f4d4da710c69f85a82c25 already in progress: creating tablet
W20240503 14:12:09.040349 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7abcb8658fc34c939aa74cf0bd981ae4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7abcb8658fc34c939aa74cf0bd981ae4 already in progress: creating tablet
W20240503 14:12:09.040630 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 50edc38cdbe74397aa4f4928d56d9fbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50edc38cdbe74397aa4f4928d56d9fbc already in progress: creating tablet
I20240503 14:12:09.043179 28661 ts_tablet_manager.cc:1875] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.047884 28653 ts_tablet_manager.cc:1888] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:09.049146 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c5297664a4d943c28adfb6ac4f061680 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5297664a4d943c28adfb6ac4f061680 already in progress: creating tablet
W20240503 14:12:09.049594 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e135b60660f448a7974520f2f87cf93a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e135b60660f448a7974520f2f87cf93a already in progress: creating tablet
I20240503 14:12:09.051097 28653 log.cc:1196] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/59a91f22268346b38ae35f96b0d763f7
I20240503 14:12:09.051340 28661 ts_tablet_manager.cc:1888] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.051963 28653 ts_tablet_manager.cc:1909] T 59a91f22268346b38ae35f96b0d763f7 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:09.049839 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e355aafb2f5b4337bedaac6fe5539dc2 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
W20240503 14:12:09.056480 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 74df7781cc6a4dd6b6fd78a728ffd71b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74df7781cc6a4dd6b6fd78a728ffd71b already in progress: creating tablet
W20240503 14:12:09.056779 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1d77014c66624f958b38f7cc83644891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d77014c66624f958b38f7cc83644891 already in progress: creating tablet
I20240503 14:12:09.057399 28600 tablet_bootstrap.cc:492] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.057673 28600 ts_tablet_manager.cc:1367] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.002s
W20240503 14:12:09.058343 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2404807f8500437ababca53f6388b636 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2404807f8500437ababca53f6388b636 already in progress: creating tablet
I20240503 14:12:09.056504 28604 tablet_bootstrap.cc:492] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.058640 28604 ts_tablet_manager.cc:1367] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.004s
W20240503 14:12:09.058759 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6dab0296a0894a63afc16ed466c591d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dab0296a0894a63afc16ed466c591d5 already in progress: creating tablet
W20240503 14:12:09.059056 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e0a92fc3833e49aaab8c5415de9bb685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0a92fc3833e49aaab8c5415de9bb685 already in progress: creating tablet
I20240503 14:12:09.058975 28600 raft_consensus.cc:348] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.059248 28600 raft_consensus.cc:374] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:12:09.059283 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6371e68ee85a47cdb0e8e21f7124d94a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6371e68ee85a47cdb0e8e21f7124d94a already in progress: creating tablet
I20240503 14:12:09.059480 28600 raft_consensus.cc:729] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
W20240503 14:12:09.059526 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 140de4d395d54b7696062ed96b8aa89c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 140de4d395d54b7696062ed96b8aa89c already in progress: creating tablet
I20240503 14:12:09.059610 28604 raft_consensus.cc:348] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:09.059762 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 962df44408484e3f8d7b7138f27c1a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 962df44408484e3f8d7b7138f27c1a66 already in progress: creating tablet
I20240503 14:12:09.059876 28604 raft_consensus.cc:374] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.059818 28600 consensus_queue.cc:260] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.060001 28604 raft_consensus.cc:729] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:09.060014 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 3703ab7db61b4c9ead3058bfdc123189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3703ab7db61b4c9ead3058bfdc123189 already in progress: creating tablet
I20240503 14:12:09.060307 28604 consensus_queue.cc:260] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.060931 28604 ts_tablet_manager.cc:1398] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:09.058389 29182 raft_consensus.cc:482] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.061371 28600 ts_tablet_manager.cc:1398] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:09.061460 29182 raft_consensus.cc:504] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.062250 28600 tablet_bootstrap.cc:492] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.062330 29182 leader_election.cc:288] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.062904 28604 tablet_bootstrap.cc:492] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.063084 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351d5bfb12c34c5f9de90e2df1b36fc1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.063418 28352 raft_consensus.cc:2454] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:09.063999 28427 leader_election.cc:302] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:09.064388 29182 raft_consensus.cc:2781] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.064538 29182 raft_consensus.cc:482] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.064561 28600 tablet_bootstrap.cc:654] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.064646 29182 raft_consensus.cc:3037] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:09.066615 28427 leader_election.cc:341] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 351d5bfb12c34c5f9de90e2df1b36fc1
I20240503 14:12:09.068171 29182 raft_consensus.cc:504] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.069039 29182 leader_election.cc:288] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.070148 28604 tablet_bootstrap.cc:654] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:09.070222 28427 leader_election.cc:341] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 351d5bfb12c34c5f9de90e2df1b36fc1
I20240503 14:12:09.071144 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351d5bfb12c34c5f9de90e2df1b36fc1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.071455 28352 raft_consensus.cc:3037] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.073161 28653 tablet_replica.cc:330] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.073432 28653 raft_consensus.cc:2229] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.073725 28653 raft_consensus.cc:2258] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.073897 28661 log.cc:1196] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ffbd3291c56b4c13813c80d6775afe6f
I20240503 14:12:09.074666 28661 ts_tablet_manager.cc:1909] T ffbd3291c56b4c13813c80d6775afe6f P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.075598 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 209) took 3070 ms. Trace:
I20240503 14:12:09.075738 28352 raft_consensus.cc:2454] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:09.075886 28661 rpcz_store.cc:276] 0503 14:12:06.005451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.005548 (+ 97us) service_pool.cc:225] Handling call
0503 14:12:09.035899 (+3030351us) ts_tablet_manager.cc:1120] Deleting tablet ffbd3291c56b4c13813c80d6775afe6f
0503 14:12:09.035910 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.047177 (+ 11267us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.051125 (+ 3948us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.075546 (+ 24421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3023483}
I20240503 14:12:09.076530 28427 leader_election.cc:302] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:09.076954 29182 raft_consensus.cc:2781] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.077333 29182 raft_consensus.cc:686] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:09.077503 29180 raft_consensus.cc:482] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.077764 28653 ts_tablet_manager.cc:1875] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.077685 29180 raft_consensus.cc:504] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.077775 29182 consensus_queue.cc:237] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.078555 29180 leader_election.cc:288] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.076601 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 216) took 3042 ms. Trace:
I20240503 14:12:09.079524 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03bf91cf5414208b6767482223bc240" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:09.079814 28212 raft_consensus.cc:2454] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:09.079481 28661 rpcz_store.cc:276] 0503 14:12:06.034095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.108308 (+ 74213us) service_pool.cc:225] Handling call
0503 14:12:09.076362 (+2968054us) ts_tablet_manager.cc:1120] Deleting tablet 0b255d3e3dc64fb69c2ef6ed389db6a6
0503 14:12:09.076374 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.076534 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2960415}
I20240503 14:12:09.080389 28427 leader_election.cc:302] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:09.080747 29180 raft_consensus.cc:2781] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.080879 29180 raft_consensus.cc:482] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.081043 29180 raft_consensus.cc:3037] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.081082 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 217) took 3046 ms. Trace:
I20240503 14:12:09.081440 28661 rpcz_store.cc:276] 0503 14:12:06.034255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.115983 (+ 81728us) service_pool.cc:225] Handling call
0503 14:12:09.080378 (+2964395us) ts_tablet_manager.cc:1120] Deleting tablet 5775006e9f8f4612aa70dfb316de9ea1
0503 14:12:09.080389 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.080889 (+ 500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2823840}
I20240503 14:12:09.082597 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 230) took 2677 ms. Trace:
I20240503 14:12:09.083341 28661 rpcz_store.cc:276] 0503 14:12:06.405048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.405135 (+ 87us) service_pool.cc:225] Handling call
0503 14:12:09.082374 (+2677239us) ts_tablet_manager.cc:1120] Deleting tablet 06fb79fcf30d485798c64c815e372dc1
0503 14:12:09.082386 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.082525 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2666960}
I20240503 14:12:09.080514 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03bf91cf5414208b6767482223bc240" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:09.084754 28427 leader_election.cc:341] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.084992 29180 raft_consensus.cc:504] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.085554 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 232) took 2657 ms. Trace:
I20240503 14:12:09.086508 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03bf91cf5414208b6767482223bc240" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.087064 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03bf91cf5414208b6767482223bc240" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:09.087337 28212 raft_consensus.cc:3037] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.087409 28661 rpcz_store.cc:276] 0503 14:12:06.428758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.428838 (+ 80us) service_pool.cc:225] Handling call
0503 14:12:09.084410 (+2655572us) ts_tablet_manager.cc:1120] Deleting tablet 65cdd090bf0f493f8b92341369378532
0503 14:12:09.084422 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.085472 (+ 1050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2631283}
I20240503 14:12:09.085965 29180 leader_election.cc:288] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:09.088207 28427 leader_election.cc:341] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.088645 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 235) took 2528 ms. Trace:
I20240503 14:12:09.088804 28661 rpcz_store.cc:276] 0503 14:12:06.560317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.560367 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:09.087734 (+2527367us) ts_tablet_manager.cc:1120] Deleting tablet 75dafd98b05d4c63bbf30165a2ac8d1e
0503 14:12:09.088412 (+ 678us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.088570 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2515987}
I20240503 14:12:09.089442 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 234) took 2537 ms. Trace:
I20240503 14:12:09.090075 28661 rpcz_store.cc:276] 0503 14:12:06.551759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.552178 (+ 419us) service_pool.cc:225] Handling call
0503 14:12:09.089252 (+2537074us) ts_tablet_manager.cc:1120] Deleting tablet b25c852aa7aa4fbfafdacec43f5bc426
0503 14:12:09.089267 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.089384 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2491943}
I20240503 14:12:09.090664 28661 tablet_replica.cc:330] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.090688 28212 raft_consensus.cc:2454] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:09.090951 28661 raft_consensus.cc:2229] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.091204 28661 raft_consensus.cc:2258] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.091928 28661 ts_tablet_manager.cc:1875] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.093573 28614 tablet_bootstrap.cc:492] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.093784 28614 ts_tablet_manager.cc:1367] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20240503 14:12:09.094888 28614 raft_consensus.cc:348] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.095191 28614 raft_consensus.cc:374] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.095331 28614 raft_consensus.cc:729] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.095700 28614 consensus_queue.cc:260] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.096385 28427 leader_election.cc:302] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:09.096504 28614 ts_tablet_manager.cc:1398] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:09.097844 29180 raft_consensus.cc:2781] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.098088 29180 raft_consensus.cc:686] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:09.098438 29180 consensus_queue.cc:237] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:09.100494 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 1d77014c66624f958b38f7cc83644891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d77014c66624f958b38f7cc83644891 already in progress: creating tablet
W20240503 14:12:09.101018 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 74df7781cc6a4dd6b6fd78a728ffd71b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74df7781cc6a4dd6b6fd78a728ffd71b already in progress: creating tablet
I20240503 14:12:09.102952 28604 tablet_bootstrap.cc:492] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.103327 28604 ts_tablet_manager.cc:1367] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.002s
I20240503 14:12:09.104637 28614 tablet_bootstrap.cc:492] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.105922 28604 raft_consensus.cc:348] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:09.108330 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2404807f8500437ababca53f6388b636 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2404807f8500437ababca53f6388b636 already in progress: creating tablet
I20240503 14:12:09.108774 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6dab0296a0894a63afc16ed466c591d5 (table default.Table_de96804c306a4b0e91a2e6bffd5943a8 [id=1b1ed3d3a0b24ced96ba66459ccecc40]) successfully deleted
I20240503 14:12:09.108937 28653 ts_tablet_manager.cc:1888] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:09.109102 28600 tablet_bootstrap.cc:492] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
W20240503 14:12:09.109122 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e0a92fc3833e49aaab8c5415de9bb685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0a92fc3833e49aaab8c5415de9bb685 already in progress: creating tablet
I20240503 14:12:09.109390 28653 log.cc:1196] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/1751f775fe6943d9b5fcc3b113393a73
W20240503 14:12:09.109393 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6371e68ee85a47cdb0e8e21f7124d94a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6371e68ee85a47cdb0e8e21f7124d94a already in progress: creating tablet
I20240503 14:12:09.109438 28600 ts_tablet_manager.cc:1367] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.005s
W20240503 14:12:09.109655 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 140de4d395d54b7696062ed96b8aa89c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 140de4d395d54b7696062ed96b8aa89c already in progress: creating tablet
I20240503 14:12:09.109696 28604 raft_consensus.cc:374] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.109880 28604 raft_consensus.cc:729] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:09.109930 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 962df44408484e3f8d7b7138f27c1a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 962df44408484e3f8d7b7138f27c1a66 already in progress: creating tablet
I20240503 14:12:09.108465 28661 ts_tablet_manager.cc:1888] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:09.110559 28661 log.cc:1196] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/8ba629915ef44e0882beee2612e285c1
I20240503 14:12:09.110658 28600 raft_consensus.cc:348] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.110677 28604 consensus_queue.cc:260] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.110973 28600 raft_consensus.cc:374] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.111084 28600 raft_consensus.cc:729] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.111358 28661 ts_tablet_manager.cc:1909] T 8ba629915ef44e0882beee2612e285c1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.111429 28604 ts_tablet_manager.cc:1398] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:09.111413 28600 consensus_queue.cc:260] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.111933 28604 tablet_bootstrap.cc:492] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.112200 28600 ts_tablet_manager.cc:1398] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:09.112488 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 233) took 2610 ms. Trace:
I20240503 14:12:09.112670 28661 rpcz_store.cc:276] 0503 14:12:06.502173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.660574 (+158401us) service_pool.cc:225] Handling call
0503 14:12:09.090564 (+2429990us) ts_tablet_manager.cc:1120] Deleting tablet 8ba629915ef44e0882beee2612e285c1
0503 14:12:09.090574 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.095908 (+ 5334us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.108304 (+ 12396us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.112416 (+ 4112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2429595}
I20240503 14:12:09.112833 28653 ts_tablet_manager.cc:1909] T 1751f775fe6943d9b5fcc3b113393a73 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:09.113440 28661 tablet_replica.cc:330] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.113952 28600 tablet_bootstrap.cc:492] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.114267 28604 tablet_bootstrap.cc:654] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.116381 28661 raft_consensus.cc:2229] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.116787 28661 raft_consensus.cc:2258] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.117692 28600 tablet_bootstrap.cc:654] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.123611 28604 tablet_bootstrap.cc:492] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.123934 28604 ts_tablet_manager.cc:1367] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
W20240503 14:12:09.123934 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3703ab7db61b4c9ead3058bfdc123189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3703ab7db61b4c9ead3058bfdc123189 already in progress: creating tablet
I20240503 14:12:09.124379 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 59a91f22268346b38ae35f96b0d763f7 (table default.Table_59af1c61c0534502b0f2cdaff98f4e96 [id=434814ed785f43e380e9c8b1cd884cf2]) successfully deleted
W20240503 14:12:09.124694 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2f89fca0480d4bf28160384ae0421bce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f89fca0480d4bf28160384ae0421bce already in progress: creating tablet
I20240503 14:12:09.125164 28661 ts_tablet_manager.cc:1875] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.125873 28604 raft_consensus.cc:348] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.122853 28603 ts_tablet_manager.cc:1888] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.126603 28603 log.cc:1196] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a0e8b0eed57c4ad1bf6040507f6f8d4c
W20240503 14:12:09.125003 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6c7c5dfac02d4dfe932996d2a37fe031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c7c5dfac02d4dfe932996d2a37fe031 already in progress: creating tablet
W20240503 14:12:09.127440 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet bf21f6234dd64cf8a32f1599fe50746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf21f6234dd64cf8a32f1599fe50746e already in progress: creating tablet
W20240503 14:12:09.127784 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 903f11a608b1483887d9b5234a404470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 903f11a608b1483887d9b5234a404470 already in progress: creating tablet
I20240503 14:12:09.128201 28603 ts_tablet_manager.cc:1909] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:09.128639 28604 raft_consensus.cc:374] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.128891 28604 raft_consensus.cc:729] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.129284 28604 consensus_queue.cc:260] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.130095 28604 ts_tablet_manager.cc:1398] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20240503 14:12:09.128033 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f45eb624434a404d85dcd233721ebf68 because the tablet was not found. No further retry: Not found: Tablet not found: f45eb624434a404d85dcd233721ebf68
W20240503 14:12:09.131314 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6d6b50aed01a4bf8af8889036dd83606 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6b50aed01a4bf8af8889036dd83606 already in progress: creating tablet
I20240503 14:12:09.131539 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 1751f775fe6943d9b5fcc3b113393a73 (table default.Table_a7c8453b862c49c6923695b05befc40b [id=9931742257d74df99c043e569a267ced]) successfully deleted
W20240503 14:12:09.131810 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b7d4e4a1f0b14406be957426a45bf45d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d4e4a1f0b14406be957426a45bf45d already in progress: creating tablet
I20240503 14:12:09.135567 28604 tablet_bootstrap.cc:492] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.137178 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 290) took 1001 ms. Trace:
I20240503 14:12:09.137351 28603 rpcz_store.cc:276] 0503 14:12:08.135267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:08.135750 (+ 483us) service_pool.cc:225] Handling call
0503 14:12:09.027547 (+891797us) ts_tablet_manager.cc:1120] Deleting tablet a0e8b0eed57c4ad1bf6040507f6f8d4c
0503 14:12:09.027555 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.089891 (+ 62336us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.122687 (+ 32796us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.137104 (+ 14417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":891459}
I20240503 14:12:09.137948 28604 tablet_bootstrap.cc:654] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.132001 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet f51834695b8b4a729a088da81eee2e47 (table default.Table_b0e228f901e4427ca9bbfc14b02c1f7f [id=743c045f2e104da8b09fd13245037eb6]) successfully deleted
W20240503 14:12:09.139083 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 4775ff1029e6449c83219030e3809984 because the tablet was not found. No further retry: Not found: Tablet not found: 4775ff1029e6449c83219030e3809984
I20240503 14:12:09.139540 28603 tablet_replica.cc:330] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:09.139808 28603 raft_consensus.cc:2229] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.140070 28603 raft_consensus.cc:2258] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.141039 28603 ts_tablet_manager.cc:1875] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.141608 28600 tablet_bootstrap.cc:492] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.141952 28600 ts_tablet_manager.cc:1367] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.003s
I20240503 14:12:09.142706 28661 ts_tablet_manager.cc:1888] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.142913 28661 log.cc:1196] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a0e8b0eed57c4ad1bf6040507f6f8d4c
I20240503 14:12:09.143476 28661 ts_tablet_manager.cc:1909] T a0e8b0eed57c4ad1bf6040507f6f8d4c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.143357 28600 raft_consensus.cc:348] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.143841 28600 raft_consensus.cc:374] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.144027 28600 raft_consensus.cc:729] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.144403 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 248) took 2518 ms. Trace:
I20240503 14:12:09.144564 28661 rpcz_store.cc:276] 0503 14:12:06.625512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:06.711590 (+ 86078us) service_pool.cc:225] Handling call
0503 14:12:09.113305 (+2401715us) ts_tablet_manager.cc:1120] Deleting tablet a0e8b0eed57c4ad1bf6040507f6f8d4c
0503 14:12:09.113317 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.131887 (+ 18570us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.142529 (+ 10642us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.144358 (+ 1829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2380713}
I20240503 14:12:09.144484 28600 consensus_queue.cc:260] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.126614 28614 tablet_bootstrap.cc:654] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.145097 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 253) took 2008 ms. Trace:
I20240503 14:12:09.145260 28661 rpcz_store.cc:276] 0503 14:12:07.136665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.136740 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:09.144898 (+2008158us) ts_tablet_manager.cc:1120] Deleting tablet 50edc38cdbe74397aa4f4928d56d9fbc
0503 14:12:09.144909 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.145042 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007657}
I20240503 14:12:09.145699 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 252) took 2011 ms. Trace:
W20240503 14:12:09.145661 27851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu pid 27927: Timed out: CheckLeaks RPC to 127.27.50.253:44259 timed out after 1.000s (SENT)
I20240503 14:12:09.145888 27851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu with pid 27927
I20240503 14:12:09.145908 28661 rpcz_store.cc:276] 0503 14:12:07.134350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.137256 (+ 2906us) service_pool.cc:225] Handling call
0503 14:12:09.145528 (+2008272us) ts_tablet_manager.cc:1120] Deleting tablet 1c5e3649c71f4d4da710c69f85a82c25
0503 14:12:09.145537 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.145645 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007996}
W20240503 14:12:09.146559 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7fe5d280a1654d80a2176c1def459e2e because the tablet was not found. No further retry: Not found: Tablet not found: 7fe5d280a1654d80a2176c1def459e2e
I20240503 14:12:09.147073 28600 ts_tablet_manager.cc:1398] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240503 14:12:09.147523 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9dbabe5beb9a42c69cb64e1b364d75a1 because the tablet was not found. No further retry: Not found: Tablet not found: 9dbabe5beb9a42c69cb64e1b364d75a1
I20240503 14:12:09.148026 28600 tablet_bootstrap.cc:492] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:09.148403 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet bacf2d1091ee458c961c691093ea65a9 because the tablet was not found. No further retry: Not found: Tablet not found: bacf2d1091ee458c961c691093ea65a9
I20240503 14:12:09.151015 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 254) took 2012 ms. Trace:
I20240503 14:12:09.151216 28661 rpcz_store.cc:276] 0503 14:12:07.138904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.139166 (+ 262us) service_pool.cc:225] Handling call
0503 14:12:09.150723 (+2011557us) ts_tablet_manager.cc:1120] Deleting tablet 7abcb8658fc34c939aa74cf0bd981ae4
0503 14:12:09.150740 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.150940 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2006623}
I20240503 14:12:09.151796 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 255) took 2011 ms. Trace:
W20240503 14:12:09.150032 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 47ba571a81eb4240a4a4a4a08cb879c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47ba571a81eb4240a4a4a4a08cb879c5 already in progress: creating tablet
I20240503 14:12:09.151957 28661 rpcz_store.cc:276] 0503 14:12:07.139944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.140068 (+ 124us) service_pool.cc:225] Handling call
0503 14:12:09.151557 (+2011489us) ts_tablet_manager.cc:1120] Deleting tablet c5297664a4d943c28adfb6ac4f061680
0503 14:12:09.151568 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.151736 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2011015}
I20240503 14:12:09.152439 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 256) took 1920 ms. Trace:
I20240503 14:12:09.152602 28661 rpcz_store.cc:276] 0503 14:12:07.232280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.232659 (+ 379us) service_pool.cc:225] Handling call
0503 14:12:09.152268 (+1919609us) ts_tablet_manager.cc:1120] Deleting tablet e135b60660f448a7974520f2f87cf93a
0503 14:12:09.152277 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.152384 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900355}
I20240503 14:12:09.152941 28661 tablet_replica.cc:330] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:09.153052 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet eb79edba14b646ef9be239b05a291d71 because the tablet was not found. No further retry: Not found: Tablet not found: eb79edba14b646ef9be239b05a291d71
I20240503 14:12:09.153218 28661 raft_consensus.cc:2229] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:09.153550 28661 pending_rounds.cc:62] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:09.153684 28661 pending_rounds.cc:68] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597683595530240 op_type: NO_OP noop_request { }
I20240503 14:12:09.153869 28661 raft_consensus.cc:2866] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:09.153872 28603 ts_tablet_manager.cc:1888] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.154039 28603 log.cc:1196] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a8d2600031df4d77b9afe636a9b5ad40
I20240503 14:12:09.154067 28661 raft_consensus.cc:2258] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.154475 28603 ts_tablet_manager.cc:1909] T a8d2600031df4d77b9afe636a9b5ad40 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:09.154778 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f50ad403bd354559b2522a76af1fa722 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50ad403bd354559b2522a76af1fa722 already in progress: creating tablet
I20240503 14:12:09.158939 28661 ts_tablet_manager.cc:1875] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.160073 28600 tablet_bootstrap.cc:654] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:09.160616 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 37dc7d96357743089372baf71654b1e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37dc7d96357743089372baf71654b1e3 already in progress: creating tablet
I20240503 14:12:09.165598 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ffbd3291c56b4c13813c80d6775afe6f (table default.Table_f99793320ab94248abb70c15ee57ca02 [id=cde896fd6d7345f998085c10bbdee418]) successfully deleted
I20240503 14:12:09.165992 28603 tablet_replica.cc:330] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:09.166297 28603 raft_consensus.cc:2229] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:09.166039 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 already in progress: creating tablet
I20240503 14:12:09.166519 28603 raft_consensus.cc:2258] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.167390 28603 ts_tablet_manager.cc:1875] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:09.167840 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5775006e9f8f4612aa70dfb316de9ea1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5775006e9f8f4612aa70dfb316de9ea1 already in progress: creating tablet
W20240503 14:12:09.168211 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 06fb79fcf30d485798c64c815e372dc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06fb79fcf30d485798c64c815e372dc1 already in progress: creating tablet
W20240503 14:12:09.169874 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 65cdd090bf0f493f8b92341369378532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65cdd090bf0f493f8b92341369378532 already in progress: creating tablet
W20240503 14:12:09.170334 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 75dafd98b05d4c63bbf30165a2ac8d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75dafd98b05d4c63bbf30165a2ac8d1e already in progress: creating tablet
W20240503 14:12:09.170678 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b25c852aa7aa4fbfafdacec43f5bc426 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25c852aa7aa4fbfafdacec43f5bc426 already in progress: creating tablet
I20240503 14:12:09.171005 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 8ba629915ef44e0882beee2612e285c1 (table default.Table_5cf8c9529d1f4adb9fbee8eb11985651 [id=bfa896d86f6a447089c787a2c220ab9e]) successfully deleted
I20240503 14:12:09.171293 28661 ts_tablet_manager.cc:1888] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.171468 28661 log.cc:1196] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e355aafb2f5b4337bedaac6fe5539dc2
I20240503 14:12:09.171705 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a0e8b0eed57c4ad1bf6040507f6f8d4c (table default.Table_4033322e04a3488cbc8be0fe6a61bec1 [id=e3f646d9aa8946529e897c83c0b10f44]) successfully deleted
I20240503 14:12:09.171941 28661 ts_tablet_manager.cc:1909] T e355aafb2f5b4337bedaac6fe5539dc2 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:09.173430 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 50edc38cdbe74397aa4f4928d56d9fbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50edc38cdbe74397aa4f4928d56d9fbc already in progress: creating tablet
W20240503 14:12:09.173852 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1c5e3649c71f4d4da710c69f85a82c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c5e3649c71f4d4da710c69f85a82c25 already in progress: creating tablet
W20240503 14:12:09.174157 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7abcb8658fc34c939aa74cf0bd981ae4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7abcb8658fc34c939aa74cf0bd981ae4 already in progress: creating tablet
W20240503 14:12:09.174418 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c5297664a4d943c28adfb6ac4f061680 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5297664a4d943c28adfb6ac4f061680 already in progress: creating tablet
W20240503 14:12:09.174667 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e135b60660f448a7974520f2f87cf93a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e135b60660f448a7974520f2f87cf93a already in progress: creating tablet
I20240503 14:12:09.177104 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 257) took 1782 ms. Trace:
I20240503 14:12:09.177282 28661 rpcz_store.cc:276] 0503 14:12:07.394731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.397056 (+ 2325us) service_pool.cc:225] Handling call
0503 14:12:09.152861 (+1755805us) ts_tablet_manager.cc:1120] Deleting tablet e355aafb2f5b4337bedaac6fe5539dc2
0503 14:12:09.152869 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.163299 (+ 10430us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.171107 (+ 7808us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.177047 (+ 5940us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1755475}
I20240503 14:12:09.177722 28661 tablet_replica.cc:330] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.177945 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a0e8b0eed57c4ad1bf6040507f6f8d4c (table default.Table_4033322e04a3488cbc8be0fe6a61bec1 [id=e3f646d9aa8946529e897c83c0b10f44]) successfully deleted
I20240503 14:12:09.177973 28661 raft_consensus.cc:2229] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.178184 28661 raft_consensus.cc:2258] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.178771 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a8d2600031df4d77b9afe636a9b5ad40 (table default.Table_22032e3385624e98a1b0bb1484f51c64 [id=304746db399b4284999a5ca79e9e80f4]) successfully deleted
I20240503 14:12:09.179309 28614 tablet_bootstrap.cc:492] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.179567 28614 ts_tablet_manager.cc:1367] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.001s
I20240503 14:12:09.180589 28614 raft_consensus.cc:348] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.181003 28614 raft_consensus.cc:374] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.181154 28614 raft_consensus.cc:729] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.181170 28661 ts_tablet_manager.cc:1875] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.181569 28614 consensus_queue.cc:260] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.184911 28603 ts_tablet_manager.cc:1888] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.185169 28603 log.cc:1196] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ea3e9c030a424740bb2fb641328adfe1
I20240503 14:12:09.183838 28614 ts_tablet_manager.cc:1398] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:09.185907 28603 ts_tablet_manager.cc:1909] T ea3e9c030a424740bb2fb641328adfe1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:09.187877 28603 tablet_replica.cc:330] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:09.188269 28603 raft_consensus.cc:2229] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.188635 28603 raft_consensus.cc:2258] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.189472 28614 tablet_bootstrap.cc:492] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.191040 28604 tablet_bootstrap.cc:492] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.191323 28604 ts_tablet_manager.cc:1367] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.015s
I20240503 14:12:09.192276 28604 raft_consensus.cc:348] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.189812 28603 ts_tablet_manager.cc:1875] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.192574 28604 raft_consensus.cc:374] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.192692 28604 raft_consensus.cc:729] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.193037 28604 consensus_queue.cc:260] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.193696 28604 ts_tablet_manager.cc:1398] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:09.193715 28614 tablet_bootstrap.cc:654] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.196126 28661 ts_tablet_manager.cc:1888] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.196626 28604 tablet_bootstrap.cc:492] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.198463 28600 tablet_bootstrap.cc:492] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.198772 28600 ts_tablet_manager.cc:1367] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.002s
I20240503 14:12:09.198796 28604 tablet_bootstrap.cc:654] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.199776 28661 log.cc:1196] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/eee989ff083041dc8dcd2cd1b220780d
I20240503 14:12:09.200320 28661 ts_tablet_manager.cc:1909] T eee989ff083041dc8dcd2cd1b220780d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.201318 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 258) took 1803 ms. Trace:
I20240503 14:12:09.201486 28661 rpcz_store.cc:276] 0503 14:12:07.397644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.397980 (+ 336us) service_pool.cc:225] Handling call
0503 14:12:09.177629 (+1779649us) ts_tablet_manager.cc:1120] Deleting tablet eee989ff083041dc8dcd2cd1b220780d
0503 14:12:09.177642 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.186585 (+ 8943us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.195925 (+ 9340us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.201256 (+ 5331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1779265}
I20240503 14:12:09.194272 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e355aafb2f5b4337bedaac6fe5539dc2 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
I20240503 14:12:09.202618 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 265) took 1733 ms. Trace:
I20240503 14:12:09.203114 28661 rpcz_store.cc:276] 0503 14:12:07.468637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.468687 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:09.202114 (+1733427us) ts_tablet_manager.cc:1120] Deleting tablet 74df7781cc6a4dd6b6fd78a728ffd71b
0503 14:12:09.202286 (+ 172us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.202536 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1733041}
I20240503 14:12:09.203687 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ea3e9c030a424740bb2fb641328adfe1 (table default.Table_3668987e06c74fa6bc566d6e46f92879 [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
I20240503 14:12:09.204123 28661 tablet_replica.cc:330] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:09.204070 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6c7c5dfac02d4dfe932996d2a37fe031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c7c5dfac02d4dfe932996d2a37fe031 already in progress: creating tablet
I20240503 14:12:09.204537 28661 raft_consensus.cc:2229] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:09.205184 28661 pending_rounds.cc:62] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:09.205263 28600 raft_consensus.cc:348] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.205626 28661 pending_rounds.cc:68] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597687105327104 op_type: NO_OP noop_request { }
I20240503 14:12:09.206115 28661 raft_consensus.cc:2866] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:09.206435 28661 raft_consensus.cc:2258] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.208598 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet eee989ff083041dc8dcd2cd1b220780d (table default.Table_706c4da08fa84bbaa207fc71eb482682 [id=36f9af99d9d44c32876688fc8cbc4f8b]) successfully deleted
I20240503 14:12:09.208752 28603 ts_tablet_manager.cc:1888] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:09.208920 28603 log.cc:1196] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/1751f775fe6943d9b5fcc3b113393a73
I20240503 14:12:09.209409 28603 ts_tablet_manager.cc:1909] T 1751f775fe6943d9b5fcc3b113393a73 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:09.209794 28600 raft_consensus.cc:374] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:12:09.210201 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 74df7781cc6a4dd6b6fd78a728ffd71b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74df7781cc6a4dd6b6fd78a728ffd71b already in progress: creating tablet
I20240503 14:12:09.210393 28600 raft_consensus.cc:729] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.211356 28661 ts_tablet_manager.cc:1875] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.211764 28600 consensus_queue.cc:260] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.212411 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 1751f775fe6943d9b5fcc3b113393a73 (table default.Table_a7c8453b862c49c6923695b05befc40b [id=9931742257d74df99c043e569a267ced]) successfully deleted
I20240503 14:12:09.213017 28600 ts_tablet_manager.cc:1398] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20240503 14:12:09.213960 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 240abf8c2b2343408fc98a8dac1fdf09 consensus state has no leader
W20240503 14:12:09.217077 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6d6b50aed01a4bf8af8889036dd83606 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6b50aed01a4bf8af8889036dd83606 already in progress: creating tablet
W20240503 14:12:09.218943 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2f89fca0480d4bf28160384ae0421bce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f89fca0480d4bf28160384ae0421bce already in progress: creating tablet
I20240503 14:12:09.220322 28604 tablet_bootstrap.cc:492] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.220536 28604 ts_tablet_manager.cc:1367] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.010s
I20240503 14:12:09.221565 28604 raft_consensus.cc:348] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.221818 28604 raft_consensus.cc:374] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.221920 28604 raft_consensus.cc:729] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.222186 28604 consensus_queue.cc:260] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.222662 28661 ts_tablet_manager.cc:1888] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.222708 28614 tablet_bootstrap.cc:492] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.222826 28604 ts_tablet_manager.cc:1398] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:09.222879 28614 ts_tablet_manager.cc:1367] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.003s
I20240503 14:12:09.222975 28661 log.cc:1196] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/351d5bfb12c34c5f9de90e2df1b36fc1
I20240503 14:12:09.223153 28604 tablet_bootstrap.cc:492] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.223595 28661 ts_tablet_manager.cc:1909] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.224262 28600 tablet_bootstrap.cc:492] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.224656 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 264) took 1756 ms. Trace:
W20240503 14:12:09.224922 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 903f11a608b1483887d9b5234a404470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 903f11a608b1483887d9b5234a404470 already in progress: creating tablet
I20240503 14:12:09.225054 28604 tablet_bootstrap.cc:654] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:09.225180 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet bf21f6234dd64cf8a32f1599fe50746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf21f6234dd64cf8a32f1599fe50746e already in progress: creating tablet
W20240503 14:12:09.225400 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b7d4e4a1f0b14406be957426a45bf45d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d4e4a1f0b14406be957426a45bf45d already in progress: creating tablet
I20240503 14:12:09.225843 28661 rpcz_store.cc:276] 0503 14:12:07.468452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.468519 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:09.203730 (+1735211us) ts_tablet_manager.cc:1120] Deleting tablet 351d5bfb12c34c5f9de90e2df1b36fc1
0503 14:12:09.203741 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.215580 (+ 11839us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.222479 (+ 6899us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:09.224591 (+ 2112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1719399}
I20240503 14:12:09.226745 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 266) took 1738 ms. Trace:
I20240503 14:12:09.226979 28661 rpcz_store.cc:276] 0503 14:12:07.487853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.487914 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:09.226483 (+1738569us) ts_tablet_manager.cc:1120] Deleting tablet 1d77014c66624f958b38f7cc83644891
0503 14:12:09.226494 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.226668 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1737025}
I20240503 14:12:09.227710 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 495190fca06a40a386c8215eb2195920 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:09.227769 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 268) took 1648 ms. Trace:
I20240503 14:12:09.227860 28661 rpcz_store.cc:276] 0503 14:12:07.579157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.590052 (+ 10895us) service_pool.cc:225] Handling call
0503 14:12:09.227611 (+1637559us) ts_tablet_manager.cc:1120] Deleting tablet 6dab0296a0894a63afc16ed466c591d5
0503 14:12:09.227621 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.227720 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636871}
I20240503 14:12:09.228927 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 269) took 1649 ms. Trace:
I20240503 14:12:09.229102 28661 rpcz_store.cc:276] 0503 14:12:07.579287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.580526 (+ 1239us) service_pool.cc:225] Handling call
0503 14:12:09.228379 (+1647853us) ts_tablet_manager.cc:1120] Deleting tablet e0a92fc3833e49aaab8c5415de9bb685
0503 14:12:09.228387 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.228503 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1625442}
I20240503 14:12:09.229709 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 267) took 1650 ms. Trace:
I20240503 14:12:09.230031 28661 rpcz_store.cc:276] 0503 14:12:07.579045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.580428 (+ 1383us) service_pool.cc:225] Handling call
0503 14:12:09.229555 (+1649127us) ts_tablet_manager.cc:1120] Deleting tablet 2404807f8500437ababca53f6388b636
0503 14:12:09.229564 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.229653 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1626382}
I20240503 14:12:09.230589 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 270) took 1616 ms. Trace:
I20240503 14:12:09.230739 28661 rpcz_store.cc:276] 0503 14:12:07.613744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.613807 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:09.230445 (+1616638us) ts_tablet_manager.cc:1120] Deleting tablet 6371e68ee85a47cdb0e8e21f7124d94a
0503 14:12:09.230453 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.230541 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1616346}
I20240503 14:12:09.231262 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 280) took 1379 ms. Trace:
I20240503 14:12:09.231397 28661 rpcz_store.cc:276] 0503 14:12:07.851840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.851905 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:09.231117 (+1379212us) ts_tablet_manager.cc:1120] Deleting tablet 962df44408484e3f8d7b7138f27c1a66
0503 14:12:09.231125 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.231217 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1378879}
I20240503 14:12:09.231824 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 279) took 1382 ms. Trace:
I20240503 14:12:09.231958 28661 rpcz_store.cc:276] 0503 14:12:07.849389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:07.922731 (+ 73342us) service_pool.cc:225] Handling call
0503 14:12:09.231669 (+1308938us) ts_tablet_manager.cc:1120] Deleting tablet 140de4d395d54b7696062ed96b8aa89c
0503 14:12:09.231679 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.231768 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308617}
I20240503 14:12:09.223783 28614 raft_consensus.cc:348] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.232192 28614 raft_consensus.cc:374] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.232430 28614 raft_consensus.cc:729] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.232583 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 284) took 1188 ms. Trace:
I20240503 14:12:09.232748 28661 rpcz_store.cc:276] 0503 14:12:08.044245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:08.049049 (+ 4804us) service_pool.cc:225] Handling call
0503 14:12:09.232431 (+1183382us) ts_tablet_manager.cc:1120] Deleting tablet 3703ab7db61b4c9ead3058bfdc123189
0503 14:12:09.232443 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:09.232524 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1183057}
I20240503 14:12:09.232873 28614 consensus_queue.cc:260] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.233280 28604 tablet_bootstrap.cc:492] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.233518 28604 ts_tablet_manager.cc:1367] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:12:09.233841 28661 tablet_replica.cc:330] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.234117 28661 raft_consensus.cc:2229] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:09.234556 28661 raft_consensus.cc:2258] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.234462 28604 raft_consensus.cc:348] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.234911 28604 raft_consensus.cc:374] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.235052 28604 raft_consensus.cc:729] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.235625 28604 consensus_queue.cc:260] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.236284 28604 ts_tablet_manager.cc:1398] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:09.236889 28661 ts_tablet_manager.cc:1875] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.237299 28604 tablet_bootstrap.cc:492] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.240346 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
W20240503 14:12:09.241087 28030 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 25) took 764 ms (client timeout 1000 ms). Trace:
W20240503 14:12:09.241245 28030 rpcz_store.cc:269] 0503 14:12:08.476312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:08.476515 (+ 203us) service_pool.cc:225] Handling call
0503 14:12:09.241027 (+764512us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20240503 14:12:09.242198 28604 tablet_bootstrap.cc:654] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.244661 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:09.244975 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:09.245467 28614 ts_tablet_manager.cc:1398] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 14:12:09.247599 28614 tablet_bootstrap.cc:492] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.245481 29180 raft_consensus.cc:482] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.248709 29180 raft_consensus.cc:504] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.249532 29180 leader_election.cc:288] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.251230 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779f373ef12a4da091309fcf247b5693" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.251523 28352 raft_consensus.cc:2454] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:09.253311 28427 leader_election.cc:302] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:09.253736 28427 leader_election.cc:341] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 779f373ef12a4da091309fcf247b5693
I20240503 14:12:09.254982 29180 raft_consensus.cc:2781] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.255146 29180 raft_consensus.cc:482] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.255242 29180 raft_consensus.cc:3037] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.258910 29180 raft_consensus.cc:504] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.260462 29180 leader_election.cc:288] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.260423 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779f373ef12a4da091309fcf247b5693" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.260713 28352 raft_consensus.cc:3037] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:09.261567 28427 leader_election.cc:341] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 779f373ef12a4da091309fcf247b5693
I20240503 14:12:09.264786 29113 raft_consensus.cc:482] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.264977 29113 raft_consensus.cc:504] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.265658 28352 raft_consensus.cc:2454] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:09.265758 29113 leader_election.cc:288] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.266884 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50ad403bd354559b2522a76af1fa722" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.267000 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50ad403bd354559b2522a76af1fa722" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:09.267606 28150 leader_election.cc:341] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:09.267903 28150 leader_election.cc:341] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.268033 28150 leader_election.cc:302] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.268198 28427 leader_election.cc:302] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:09.268561 29182 raft_consensus.cc:2781] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.268560 29113 raft_consensus.cc:2726] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.268748 29182 raft_consensus.cc:686] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:09.269073 29182 consensus_queue.cc:237] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.277276 29113 raft_consensus.cc:482] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.277499 29113 raft_consensus.cc:504] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.248473 28661 ts_tablet_manager.cc:1888] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:09.278267 29113 leader_election.cc:288] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.278329 28661 log.cc:1196] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/1751f775fe6943d9b5fcc3b113393a73
I20240503 14:12:09.279341 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae4969fc9474fdbaeae1b4d09e4875b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.279374 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae4969fc9474fdbaeae1b4d09e4875b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.279651 28489 raft_consensus.cc:2454] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
W20240503 14:12:09.279973 28150 leader_election.cc:341] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.280283 28150 leader_election.cc:302] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:09.280879 29113 raft_consensus.cc:2781] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.280999 29113 raft_consensus.cc:482] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.281085 29113 raft_consensus.cc:3037] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.283123 28661 ts_tablet_manager.cc:1909] T 1751f775fe6943d9b5fcc3b113393a73 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.284685 29113 raft_consensus.cc:504] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.285949 29113 leader_election.cc:288] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.286136 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae4969fc9474fdbaeae1b4d09e4875b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.286340 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae4969fc9474fdbaeae1b4d09e4875b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:09.286602 28489 raft_consensus.cc:3037] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:09.286864 28150 leader_election.cc:341] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.289992 28489 raft_consensus.cc:2454] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:09.290612 28150 leader_election.cc:302] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:09.291031 29113 raft_consensus.cc:482] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.291435 29113 raft_consensus.cc:504] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.292413 28661 tablet_replica.cc:330] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.292716 28661 raft_consensus.cc:2229] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:09.293027 28661 pending_rounds.cc:62] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:09.293126 28661 pending_rounds.cc:68] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597686250225664 op_type: NO_OP noop_request { }
I20240503 14:12:09.293294 28661 raft_consensus.cc:2866] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:09.293433 28661 raft_consensus.cc:2258] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.296252 28661 ts_tablet_manager.cc:1875] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:09.296600 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1d77014c66624f958b38f7cc83644891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d77014c66624f958b38f7cc83644891 already in progress: creating tablet
W20240503 14:12:09.296942 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6dab0296a0894a63afc16ed466c591d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dab0296a0894a63afc16ed466c591d5 already in progress: creating tablet
W20240503 14:12:09.297178 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e0a92fc3833e49aaab8c5415de9bb685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0a92fc3833e49aaab8c5415de9bb685 already in progress: creating tablet
I20240503 14:12:09.297384 29113 leader_election.cc:288] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:09.297413 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2404807f8500437ababca53f6388b636 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2404807f8500437ababca53f6388b636 already in progress: creating tablet
W20240503 14:12:09.297598 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6371e68ee85a47cdb0e8e21f7124d94a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6371e68ee85a47cdb0e8e21f7124d94a already in progress: creating tablet
W20240503 14:12:09.297845 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 962df44408484e3f8d7b7138f27c1a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 962df44408484e3f8d7b7138f27c1a66 already in progress: creating tablet
W20240503 14:12:09.298066 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 140de4d395d54b7696062ed96b8aa89c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 140de4d395d54b7696062ed96b8aa89c already in progress: creating tablet
I20240503 14:12:09.297883 29113 raft_consensus.cc:2781] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
W20240503 14:12:09.298286 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 3703ab7db61b4c9ead3058bfdc123189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3703ab7db61b4c9ead3058bfdc123189 already in progress: creating tablet
W20240503 14:12:09.298497 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2f89fca0480d4bf28160384ae0421bce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f89fca0480d4bf28160384ae0421bce already in progress: creating tablet
I20240503 14:12:09.298506 29113 raft_consensus.cc:686] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
W20240503 14:12:09.298733 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6c7c5dfac02d4dfe932996d2a37fe031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c7c5dfac02d4dfe932996d2a37fe031 already in progress: creating tablet
I20240503 14:12:09.299099 29113 consensus_queue.cc:237] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.300149 28614 tablet_bootstrap.cc:654] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.301093 29200 tablet.cc:1722] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:09.305997 29204 tablet.cc:1722] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:09.309430 29202 tablet.cc:1722] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:09.312132 28290 connection.cc:535] client connection to 127.27.50.253:44259 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:09.313427 28427 connection.cc:535] client connection to 127.27.50.253:44259 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:09.318511 28600 tablet_bootstrap.cc:654] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.321695 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.322432 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.324429 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_32b3856710f446139fa555cd42254eef" } modify_external_catalogs: true
I20240503 14:12:09.324682 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_32b3856710f446139fa555cd42254eef" } modify_external_catalogs: true
I20240503 14:12:09.325394 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a2bb32853fa94122b1e4188e3fc4f65e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.325414 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 7abcb8658fc34c939aa74cf0bd981ae4 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:09.326058 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2bb32853fa94122b1e4188e3fc4f65e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.326260 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet d93fea5caec94695a5df26d3630c50c7 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:09.326412 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 93462d2b1a474bd688d6266cd3b0695c from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:09.326841 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 80cd09e769be4bee95bbef9c3c550451 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:09.328867 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:09.329252 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet e9609112657f427cbeeb3ef8f28cddf0 (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]). Expected version 5 got 0
I20240503 14:12:09.329500 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:09.329752 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]). Expected version 2 got 0
I20240503 14:12:09.331049 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_253d4e5a90d94897874db169bc3718b6" } modify_external_catalogs: true
I20240503 14:12:09.331326 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_253d4e5a90d94897874db169bc3718b6" } modify_external_catalogs: true
I20240503 14:12:09.333909 28028 catalog_manager.cc:5484] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:09.334627 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:09.335021 28028 catalog_manager.cc:5484] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:09.335546 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:09.335892 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet e9609112657f427cbeeb3ef8f28cddf0 (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]). Expected version 5 got 0
I20240503 14:12:09.337980 28028 catalog_manager.cc:5484] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:09.338805 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
W20240503 14:12:09.344517 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 53937.3us
W20240503 14:12:09.345134 28150 connection.cc:535] client connection to 127.27.50.253:44259 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:09.345500 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9609112657f427cbeeb3ef8f28cddf0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.345778 28489 raft_consensus.cc:2454] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:09.346879 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9609112657f427cbeeb3ef8f28cddf0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:09.349002 29246 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:12:09.314451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.50.253:44259 (local address 127.27.50.195:45469)
0503 14:12:09.348507 (+ 34056us) negotiation.cc:107] Waiting for socket to connect
0503 14:12:09.348607 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33931,"threads_started":1}
I20240503 14:12:09.354724 28661 ts_tablet_manager.cc:1888] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.354943 28661 log.cc:1196] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a8d2600031df4d77b9afe636a9b5ad40
I20240503 14:12:09.355434 28661 ts_tablet_manager.cc:1909] T a8d2600031df4d77b9afe636a9b5ad40 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.355527 28614 tablet_bootstrap.cc:492] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.355741 28614 ts_tablet_manager.cc:1367] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.108s user 0.014s sys 0.005s
W20240503 14:12:09.357205 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:12:09.313167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.50.253:44259 (local address 127.27.50.194:58511)
0503 14:12:09.347870 (+ 34703us) negotiation.cc:107] Waiting for socket to connect
0503 14:12:09.348011 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34582,"thread_start_us":37188,"threads_started":1}
I20240503 14:12:09.358099 28614 raft_consensus.cc:348] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.358418 28614 raft_consensus.cc:374] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.358501 28614 raft_consensus.cc:729] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.358783 28614 consensus_queue.cc:260] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.362320 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 1751f775fe6943d9b5fcc3b113393a73 (table default.Table_a7c8453b862c49c6923695b05befc40b [id=9931742257d74df99c043e569a267ced]) successfully deleted
W20240503 14:12:09.362614 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 63661.8us
I20240503 14:12:09.363413 28614 ts_tablet_manager.cc:1398] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:12:09.363773 28193 tablet_service.cc:1467] Processing CreateTablet for tablet a703cfbaad2a4a618abb9e07b4930cee (DEFAULT_TABLE table=default.Table_aa516ed89b714cf788cdf47928d6a415 [id=16cce63b7e0b4afbacc9feffaa625932]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.364249 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a703cfbaad2a4a618abb9e07b4930cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.367240 28600 tablet_bootstrap.cc:492] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.367491 28600 ts_tablet_manager.cc:1367] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.143s user 0.004s sys 0.000s
I20240503 14:12:09.367888 28614 tablet_bootstrap.cc:492] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.369097 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.369104 28474 tablet_service.cc:1467] Processing CreateTablet for tablet a703cfbaad2a4a618abb9e07b4930cee (DEFAULT_TABLE table=default.Table_aa516ed89b714cf788cdf47928d6a415 [id=16cce63b7e0b4afbacc9feffaa625932]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.369570 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a703cfbaad2a4a618abb9e07b4930cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.369578 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d54cc5c03034a6685c1da34f9bd65ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.369904 28326 tablet_service.cc:1467] Processing CreateTablet for tablet a703cfbaad2a4a618abb9e07b4930cee (DEFAULT_TABLE table=default.Table_aa516ed89b714cf788cdf47928d6a415 [id=16cce63b7e0b4afbacc9feffaa625932]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.370311 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a703cfbaad2a4a618abb9e07b4930cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.377039 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 (DEFAULT_TABLE table=default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0 [id=82ad92c83a0c44bc86486fb6137a47e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.377122 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.377241 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 (DEFAULT_TABLE table=default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0 [id=82ad92c83a0c44bc86486fb6137a47e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.377544 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d54cc5c03034a6685c1da34f9bd65ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.377687 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a50b3e199314bfc8465fdb7b2425d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.377712 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a50b3e199314bfc8465fdb7b2425d07. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:09.384307 28150 leader_election.cc:341] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.384804 28150 leader_election.cc:302] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:09.384764 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.385205 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d54cc5c03034a6685c1da34f9bd65ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.385710 29113 raft_consensus.cc:2781] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.385824 29113 raft_consensus.cc:482] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.385941 29113 raft_consensus.cc:3037] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.388399 29243 raft_consensus.cc:482] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.388465 29113 raft_consensus.cc:504] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.388579 29243 raft_consensus.cc:504] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.389310 29243 leader_election.cc:288] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.389250 29113 leader_election.cc:288] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:09.390125 28017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.50.253:44259 (error 108) [suppressed 6 similar messages]
I20240503 14:12:09.390102 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (DEFAULT_TABLE table=default.Table_027b06bfe8c444dcba6cd5d6f42220ec [id=134086bbf5ef4b5e890cab53615e449e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.390156 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9609112657f427cbeeb3ef8f28cddf0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.390604 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69425d29d55b4fe4bb7f08ae390aaa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.390967 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 42c3868ce2744d349a23779b2a62bb32 (DEFAULT_TABLE table=default.Table_adc1bb9599f646e2bf55dcf13fdf6688 [id=4dc8d8a7dfcb478388125a2d2763c249]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.391103 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.391400 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c3868ce2744d349a23779b2a62bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.391820 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 42c3868ce2744d349a23779b2a62bb32 (DEFAULT_TABLE table=default.Table_adc1bb9599f646e2bf55dcf13fdf6688 [id=4dc8d8a7dfcb478388125a2d2763c249]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:09.391935 28150 leader_election.cc:341] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.392232 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c3868ce2744d349a23779b2a62bb32. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:09.392239 28150 leader_election.cc:341] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.392410 28195 tablet_service.cc:1467] Processing CreateTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d (DEFAULT_TABLE table=default.Table_7fdf02629122402c9cbf8053f89eced3 [id=5201e94e1d164116b74d4329e5991eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.392791 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0175b38c1e0464a9a6208cfaf68ca0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.392890 28614 tablet_bootstrap.cc:654] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.396299 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbd3291c56b4c13813c80d6775afe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.396605 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 1693dd36a3d84d88a49690c537590083 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7c5dfac02d4dfe932996d2a37fe031) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.396714 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet f50ad403bd354559b2522a76af1fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.397037 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9609112657f427cbeeb3ef8f28cddf0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:09.397037 28187 tablet_service.cc:1467] Processing CreateTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d (DEFAULT_TABLE table=default.Table_7c4e797497e74bbe98ff720b9ffd05ff [id=ee4e5312119d414aa8f8d01c28e0b5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.397276 28489 raft_consensus.cc:3037] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.397456 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcc6cef7b7d464a9fbbae61caf0437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.400312 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.400456 28179 tablet_service.cc:1467] Processing CreateTablet for tablet c9325d5142af4b018354bf38370b61e0 (DEFAULT_TABLE table=default.Table_965ce30debbf4cc6a42b1fee93fb54de [id=db393a7e2a83466385ffc0af3d2789c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.400044 28489 raft_consensus.cc:2454] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:09.400890 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9325d5142af4b018354bf38370b61e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:09.400943 28150 leader_election.cc:341] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.401115 28150 leader_election.cc:302] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.401574 28150 leader_election.cc:302] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:09.401847 29243 raft_consensus.cc:2726] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.402091 29243 raft_consensus.cc:2781] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.402243 29243 raft_consensus.cc:686] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:09.402585 29243 consensus_queue.cc:237] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.403759 28456 tablet_service.cc:1467] Processing CreateTablet for tablet a884cf4ea11e48ef941722b79ad50075 (DEFAULT_TABLE table=default.Table_cc77503fb1ad4d03a6278aef9f3ab979 [id=4b3b8bed2fba4370a246902bec7554f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.404223 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a884cf4ea11e48ef941722b79ad50075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.404244 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.404525 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.404762 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e977f7837e49fb93ef0fd18b619792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.404992 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8411f474c247c2abfd0bd9682fb8a2) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.405206 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 40da5f5fff94462d8af740f3ee337e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.408548 28653 tablet_replica.cc:330] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.408808 28653 raft_consensus.cc:2229] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.409015 28653 raft_consensus.cc:2258] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.409638 28653 ts_tablet_manager.cc:1875] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.413138 28614 tablet_bootstrap.cc:492] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.413552 28614 ts_tablet_manager.cc:1367] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.009s
I20240503 14:12:09.414454 28614 raft_consensus.cc:348] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.414717 28614 raft_consensus.cc:374] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.414808 28614 raft_consensus.cc:729] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.415153 28614 consensus_queue.cc:260] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.416538 28614 ts_tablet_manager.cc:1398] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:09.417073 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 42c3868ce2744d349a23779b2a62bb32 (DEFAULT_TABLE table=default.Table_adc1bb9599f646e2bf55dcf13fdf6688 [id=4dc8d8a7dfcb478388125a2d2763c249]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.417341 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet e355aafb2f5b4337bedaac6fe5539dc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 80cd09e769be4bee95bbef9c3c550451) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.417624 28336 tablet_service.cc:1467] Processing CreateTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d (DEFAULT_TABLE table=default.Table_7fdf02629122402c9cbf8053f89eced3 [id=5201e94e1d164116b74d4329e5991eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.417824 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 (DEFAULT_TABLE table=default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0 [id=82ad92c83a0c44bc86486fb6137a47e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.418011 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (DEFAULT_TABLE table=default.Table_027b06bfe8c444dcba6cd5d6f42220ec [id=134086bbf5ef4b5e890cab53615e449e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.418802 28328 tablet_service.cc:1467] Processing CreateTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d (DEFAULT_TABLE table=default.Table_7c4e797497e74bbe98ff720b9ffd05ff [id=ee4e5312119d414aa8f8d01c28e0b5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.418908 28460 tablet_service.cc:1467] Processing CreateTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d (DEFAULT_TABLE table=default.Table_7c4e797497e74bbe98ff720b9ffd05ff [id=ee4e5312119d414aa8f8d01c28e0b5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.418964 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (DEFAULT_TABLE table=default.Table_027b06bfe8c444dcba6cd5d6f42220ec [id=134086bbf5ef4b5e890cab53615e449e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.419196 28337 tablet_service.cc:1467] Processing CreateTablet for tablet c9325d5142af4b018354bf38370b61e0 (DEFAULT_TABLE table=default.Table_965ce30debbf4cc6a42b1fee93fb54de [id=db393a7e2a83466385ffc0af3d2789c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.419368 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcc6cef7b7d464a9fbbae61caf0437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.419587 28466 tablet_service.cc:1467] Processing CreateTablet for tablet c9325d5142af4b018354bf38370b61e0 (DEFAULT_TABLE table=default.Table_965ce30debbf4cc6a42b1fee93fb54de [id=db393a7e2a83466385ffc0af3d2789c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.419951 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9325d5142af4b018354bf38370b61e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.420158 29185 raft_consensus.cc:482] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.420311 29185 raft_consensus.cc:504] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.421036 29185 leader_election.cc:288] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.417150 28334 tablet_service.cc:1467] Processing CreateTablet for tablet a884cf4ea11e48ef941722b79ad50075 (DEFAULT_TABLE table=default.Table_cc77503fb1ad4d03a6278aef9f3ab979 [id=4b3b8bed2fba4370a246902bec7554f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.422607 28614 tablet_bootstrap.cc:492] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.423395 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c3868ce2744d349a23779b2a62bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.424677 28614 tablet_bootstrap.cc:654] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.426661 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69425d29d55b4fe4bb7f08ae390aaa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.427196 28653 ts_tablet_manager.cc:1888] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.427349 28653 log.cc:1196] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f50ad403bd354559b2522a76af1fa722
I20240503 14:12:09.427697 28653 ts_tablet_manager.cc:1909] T f50ad403bd354559b2522a76af1fa722 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:09.428480 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69425d29d55b4fe4bb7f08ae390aaa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.390938 28600 raft_consensus.cc:348] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.432549 28600 raft_consensus.cc:374] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.432946 28600 raft_consensus.cc:729] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.433794 28600 consensus_queue.cc:260] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.434038 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9325d5142af4b018354bf38370b61e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.432849 28653 tablet_replica.cc:330] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.434372 28653 raft_consensus.cc:2229] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.433406 28178 tablet_service.cc:1467] Processing CreateTablet for tablet a884cf4ea11e48ef941722b79ad50075 (DEFAULT_TABLE table=default.Table_cc77503fb1ad4d03a6278aef9f3ab979 [id=4b3b8bed2fba4370a246902bec7554f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.434751 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a884cf4ea11e48ef941722b79ad50075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.434768 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcc6cef7b7d464a9fbbae61caf0437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.435243 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a884cf4ea11e48ef941722b79ad50075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.435650 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0175b38c1e0464a9a6208cfaf68ca0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.436256 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a50b3e199314bfc8465fdb7b2425d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.436412 29180 raft_consensus.cc:482] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.436575 29180 raft_consensus.cc:504] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.437234 29180 leader_election.cc:288] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.438072 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40da5f5fff94462d8af740f3ee337e76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:09.438325 28212 raft_consensus.cc:2454] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:09.439316 28653 raft_consensus.cc:2258] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.439961 28653 ts_tablet_manager.cc:1875] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.441125 28427 leader_election.cc:302] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:09.440495 28600 ts_tablet_manager.cc:1398] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.073s user 0.003s sys 0.000s
I20240503 14:12:09.441438 29180 raft_consensus.cc:2781] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.441546 29180 raft_consensus.cc:482] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.441628 29180 raft_consensus.cc:3037] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.441745 28600 tablet_bootstrap.cc:492] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.445765 29180 raft_consensus.cc:504] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.446480 29180 leader_election.cc:288] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.447304 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40da5f5fff94462d8af740f3ee337e76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:09.448225 28600 tablet_bootstrap.cc:654] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.448247 28212 raft_consensus.cc:3037] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.451265 28212 raft_consensus.cc:2454] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:09.452760 28604 tablet_bootstrap.cc:492] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.452940 28604 ts_tablet_manager.cc:1367] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.216s user 0.045s sys 0.058s
I20240503 14:12:09.453799 28604 raft_consensus.cc:348] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.454057 28604 raft_consensus.cc:374] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.454145 28604 raft_consensus.cc:729] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.454392 28604 consensus_queue.cc:260] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.454977 28604 ts_tablet_manager.cc:1398] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:09.455358 28604 tablet_bootstrap.cc:492] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.457406 28604 tablet_bootstrap.cc:654] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.457720 28653 ts_tablet_manager.cc:1888] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.457864 28653 log.cc:1196] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/75dafd98b05d4c63bbf30165a2ac8d1e
W20240503 14:12:09.458436 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 68279.9us
I20240503 14:12:09.459013 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a8d2600031df4d77b9afe636a9b5ad40 (table default.Table_22032e3385624e98a1b0bb1484f51c64 [id=304746db399b4284999a5ca79e9e80f4]) successfully deleted
I20240503 14:12:09.459136 28653 ts_tablet_manager.cc:1909] T 75dafd98b05d4c63bbf30165a2ac8d1e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:09.459323 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6d6b50aed01a4bf8af8889036dd83606 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6b50aed01a4bf8af8889036dd83606 already in progress: creating tablet
W20240503 14:12:09.459587 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 903f11a608b1483887d9b5234a404470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 903f11a608b1483887d9b5234a404470 already in progress: creating tablet
W20240503 14:12:09.459827 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet bf21f6234dd64cf8a32f1599fe50746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf21f6234dd64cf8a32f1599fe50746e already in progress: creating tablet
W20240503 14:12:09.460041 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b7d4e4a1f0b14406be957426a45bf45d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d4e4a1f0b14406be957426a45bf45d already in progress: creating tablet
I20240503 14:12:09.462134 28427 leader_election.cc:302] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:09.462293 28653 tablet_replica.cc:330] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.462456 29180 raft_consensus.cc:2781] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.462515 28653 raft_consensus.cc:2229] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:09.462594 29180 raft_consensus.cc:686] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:09.462778 28653 pending_rounds.cc:62] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:09.462852 28653 pending_rounds.cc:68] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597688158707712 op_type: NO_OP noop_request { }
I20240503 14:12:09.462983 28653 raft_consensus.cc:2866] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:09.462884 29180 consensus_queue.cc:237] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.463091 28653 raft_consensus.cc:2258] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.466807 28604 tablet_bootstrap.cc:492] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.467002 28604 ts_tablet_manager.cc:1367] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:12:09.467296 28456 tablet_service.cc:1467] Processing CreateTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d (DEFAULT_TABLE table=default.Table_7fdf02629122402c9cbf8053f89eced3 [id=5201e94e1d164116b74d4329e5991eca]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:09.469349 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ffbd3291c56b4c13813c80d6775afe6f because the tablet was not found. No further retry: Not found: Tablet not found: ffbd3291c56b4c13813c80d6775afe6f
I20240503 14:12:09.469436 28653 ts_tablet_manager.cc:1875] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.469956 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0175b38c1e0464a9a6208cfaf68ca0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.467854 28604 raft_consensus.cc:348] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.480309 28604 raft_consensus.cc:374] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.480464 28604 raft_consensus.cc:729] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.480806 28604 consensus_queue.cc:260] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.481577 28604 ts_tablet_manager.cc:1398] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:12:09.471670 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2331aec871824a238f3f64de217d7092
W20240503 14:12:09.482069 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 1693dd36a3d84d88a49690c537590083 because the tablet was not found. No further retry: Not found: Tablet not found: 1693dd36a3d84d88a49690c537590083
I20240503 14:12:09.482429 28024 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 398) took 1029 ms. Trace:
I20240503 14:12:09.482836 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet f50ad403bd354559b2522a76af1fa722 (table default.Table_5eec9210201643c3a011af892345d421 [id=f4c76f2fa96d41c491fb0ac30f49e43f]) successfully deleted
I20240503 14:12:09.483173 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 75dafd98b05d4c63bbf30165a2ac8d1e (table default.Table_986d778bcd814c9c87dbd18d1f485cc9 [id=5963eb580afa4dc18cf4e472ed1fb76c]) successfully deleted
I20240503 14:12:09.482760 28024 rpcz_store.cc:276] 0503 14:12:08.453305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:08.453391 (+ 86us) service_pool.cc:225] Handling call
0503 14:12:08.454386 (+ 995us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:08.454542 (+ 156us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:08.454636 (+ 94us) catalog_manager.cc:2758] Locking tablets
0503 14:12:09.238780 (+784144us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:09.471452 (+232672us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:09.471568 (+ 116us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:09.471616 (+ 48us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:09.482358 (+ 10742us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:09.294178 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:09.294327 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:09.294335 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:09.294337 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:09.294517 (+ 180us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:09.294531 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:09.294536 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:09.294758 (+ 222us) tablet.cc:699] Row locks acquired
0503 14:12:09.294761 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:09.294814 (+ 53us) write_op.cc:270] Start()
0503 14:12:09.294855 (+ 41us) write_op.cc:276] Timestamp: P: 1714745529294782 usec, L: 0
0503 14:12:09.294859 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:09.335884 (+ 41025us) log.cc:842] Serialized 731 byte log entry
0503 14:12:09.336565 (+ 681us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 000000000095519a 00007fe125139162 00007fe1264a1e32 00007fe1264a0959 00007fe1264a886b 00007fe1264a86e8 0000000000a60bcc 00007fe11993ab74 00007fe11993e19b 00007fe11993e018 0000000000a60bcc
0503 14:12:09.469391 (+132826us) op_driver.cc:444] REPLICATION: finished
0503 14:12:09.469581 (+ 190us) write_op.cc:301] APPLY: starting
0503 14:12:09.470628 (+ 1047us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:09.470684 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:09.470686 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:09.470768 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:09.470812 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:09.470821 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:09.470934 (+ 113us) log.cc:842] Serialized 29 byte log entry
0503 14:12:09.471012 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:09.471038 (+ 26us) write_op.cc:454] Released schema lock
0503 14:12:09.471046 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:09.471108 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:09.471112 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"apply.run_cpu_time_us":591,"apply.run_wall_time_us":1593,"num_ops":2,"prepare.queue_time_us":54475,"prepare.run_cpu_time_us":1406,"prepare.run_wall_time_us":42526,"raft.queue_time_us":1345,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":252,"replication_time_us":174499,"spinlock_wait_cycles":88086784,"thread_start_us":55323,"threads_started":1}]]}
I20240503 14:12:09.483732 28604 tablet_bootstrap.cc:492] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.483804 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_52932a5c26744566bfc744e2ed1d8aee" } modify_external_catalogs: true
I20240503 14:12:09.484001 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_52932a5c26744566bfc744e2ed1d8aee" } modify_external_catalogs: true
I20240503 14:12:09.485394 28604 tablet_bootstrap.cc:654] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.485353 28653 ts_tablet_manager.cc:1888] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.485503 28653 log.cc:1196] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/cae4969fc9474fdbaeae1b4d09e4875b
I20240503 14:12:09.485985 28653 ts_tablet_manager.cc:1909] T cae4969fc9474fdbaeae1b4d09e4875b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:09.487286 28017 consensus_peers.cc:485] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 -> Peer a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259): Couldn't send request to peer a10ddca38c26462eb221d6026e88c6cc. Status: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:09.490105 28653 tablet_replica.cc:330] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.490386 28653 raft_consensus.cc:2229] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.490592 28653 raft_consensus.cc:2258] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.491183 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40da5f5fff94462d8af740f3ee337e76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.491930 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40da5f5fff94462d8af740f3ee337e76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.492156 28653 ts_tablet_manager.cc:1875] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:09.494340 28427 leader_election.cc:341] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:09.494813 28427 leader_election.cc:341] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:09.495271 28290 leader_election.cc:341] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: ffbd3291c56b4c13813c80d6775afe6f
W20240503 14:12:09.495515 28290 leader_election.cc:341] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: ffbd3291c56b4c13813c80d6775afe6f
I20240503 14:12:09.495631 28290 leader_election.cc:302] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.495925 29185 raft_consensus.cc:2726] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.498920 28653 ts_tablet_manager.cc:1888] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.499118 28653 log.cc:1196] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/63e977f7837e49fb93ef0fd18b619792
I20240503 14:12:09.499480 28653 ts_tablet_manager.cc:1909] T 63e977f7837e49fb93ef0fd18b619792 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:09.500983 28600 tablet_bootstrap.cc:492] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.501253 28600 ts_tablet_manager.cc:1367] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.011s
W20240503 14:12:09.502604 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e355aafb2f5b4337bedaac6fe5539dc2 because the tablet was not found. No further retry: Not found: Tablet not found: e355aafb2f5b4337bedaac6fe5539dc2
I20240503 14:12:09.503700 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet cae4969fc9474fdbaeae1b4d09e4875b (table default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f [id=837e8a35f3024ddcbe19763625d82a1f]) successfully deleted
I20240503 14:12:09.503744 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 2331aec871824a238f3f64de217d7092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.504241 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 495190fca06a40a386c8215eb2195920
I20240503 14:12:09.504715 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 495190fca06a40a386c8215eb2195920 deleted and replaced by tablet 7261088abbc641db8d737a48bbac25bd
I20240503 14:12:09.505195 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8fe3affd0d144b1fb0078e7690b9b36d
I20240503 14:12:09.505995 29180 raft_consensus.cc:482] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.505980 28600 raft_consensus.cc:348] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.506244 28600 raft_consensus.cc:374] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.506330 28600 raft_consensus.cc:729] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.506199 29180 raft_consensus.cc:504] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.506604 28600 consensus_queue.cc:260] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.506883 29180 leader_election.cc:288] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.507330 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d93fea5caec94695a5df26d3630c50c7
I20240503 14:12:09.507588 28600 ts_tablet_manager.cc:1398] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:09.507813 28025 catalog_manager.cc:6554] ReplaceTablet: tablet d93fea5caec94695a5df26d3630c50c7 deleted and replaced by tablet b476e87faea1415d95ae014a93fa22af
I20240503 14:12:09.508000 28600 tablet_bootstrap.cc:492] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.508809 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 2331aec871824a238f3f64de217d7092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.509733 28600 tablet_bootstrap.cc:654] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.509843 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50ad403bd354559b2522a76af1fa722" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.510491 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 2331aec871824a238f3f64de217d7092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:09.510950 28427 leader_election.cc:341] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:09.511212 28427 leader_election.cc:341] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: f50ad403bd354559b2522a76af1fa722
I20240503 14:12:09.511349 28427 leader_election.cc:302] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:09.512750 29180 raft_consensus.cc:2726] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.515902 28604 tablet_bootstrap.cc:492] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
W20240503 14:12:09.515988 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2331aec871824a238f3f64de217d7092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2331aec871824a238f3f64de217d7092 already in progress: creating tablet
I20240503 14:12:09.516103 28604 ts_tablet_manager.cc:1367] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240503 14:12:09.516727 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 495190fca06a40a386c8215eb2195920 with delete_type TABLET_DATA_DELETED (replaced by tablet 7261088abbc641db8d737a48bbac25bd) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.516876 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 495190fca06a40a386c8215eb2195920 with delete_type TABLET_DATA_DELETED (replaced by tablet 7261088abbc641db8d737a48bbac25bd) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.516990 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.517015 28604 raft_consensus.cc:348] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.517201 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet d93fea5caec94695a5df26d3630c50c7 with delete_type TABLET_DATA_DELETED (replaced by tablet b476e87faea1415d95ae014a93fa22af) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.517256 28604 raft_consensus.cc:374] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.517361 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.517766 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet d93fea5caec94695a5df26d3630c50c7 with delete_type TABLET_DATA_DELETED (replaced by tablet b476e87faea1415d95ae014a93fa22af) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.518057 28653 tablet_replica.cc:330] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.518296 28653 raft_consensus.cc:2229] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.518496 28653 raft_consensus.cc:2258] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.519071 28653 ts_tablet_manager.cc:1875] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.520153 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 80cd09e769be4bee95bbef9c3c550451
I20240503 14:12:09.520520 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b2bba6a3d6e84552b027f23a29cbb38a
I20240503 14:12:09.520686 28026 catalog_manager.cc:6554] ReplaceTablet: tablet 80cd09e769be4bee95bbef9c3c550451 deleted and replaced by tablet 49445f0f1dd1451c9ba066e61273f2f1
I20240503 14:12:09.522120 28604 raft_consensus.cc:729] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.522452 28604 consensus_queue.cc:260] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.523051 28604 ts_tablet_manager.cc:1398] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 14:12:09.525192 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 63e977f7837e49fb93ef0fd18b619792 (table default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4 [id=dcdb4c8fe4774779b0303db262dbfcc8]) successfully deleted
W20240503 14:12:09.525609 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 495190fca06a40a386c8215eb2195920 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 495190fca06a40a386c8215eb2195920 already in progress: creating tablet
I20240503 14:12:09.525652 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cd09e769be4bee95bbef9c3c550451 with delete_type TABLET_DATA_DELETED (replaced by tablet 49445f0f1dd1451c9ba066e61273f2f1) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:09.525867 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 8fe3affd0d144b1fb0078e7690b9b36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fe3affd0d144b1fb0078e7690b9b36d already in progress: creating tablet
I20240503 14:12:09.525918 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.526136 28032 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:09.526298 29243 raft_consensus.cc:482] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.526464 29243 raft_consensus.cc:504] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:09.526777 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d93fea5caec94695a5df26d3630c50c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d93fea5caec94695a5df26d3630c50c7 already in progress: creating tablet
I20240503 14:12:09.527199 29243 leader_election.cc:288] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.527951 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:09.528729 28150 leader_election.cc:341] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.529026 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:09.530738 28150 leader_election.cc:341] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.530897 28150 leader_election.cc:302] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.530750 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.531347 29243 raft_consensus.cc:2726] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.531813 28604 tablet_bootstrap.cc:492] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.541460 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_28594db96b8b48b9af52ccdd7879ff83" } new_table_name: "default.Table_130e64b09fbf49ceac42c4f3bac8c0ba" modify_external_catalogs: true
I20240503 14:12:09.542553 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cd09e769be4bee95bbef9c3c550451 with delete_type TABLET_DATA_DELETED (replaced by tablet 49445f0f1dd1451c9ba066e61273f2f1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.542824 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.542999 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a91f22268346b38ae35f96b0d763f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.541460 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3f5415973b9641a3a612e9236aad1e92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.543551 28661 tablet_replica.cc:330] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.543787 28661 raft_consensus.cc:2229] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.543951 28661 raft_consensus.cc:2258] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.544391 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 779f373ef12a4da091309fcf247b5693 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.545132 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_42bdb5d25c3e4d778e5c09ceb819bec4" } new_table_name: "default.Table_db04b673d60644559c17717f39eab7e2" modify_external_catalogs: true
I20240503 14:12:09.545197 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7fdf02629122402c9cbf8053f89eced3" } modify_external_catalogs: true
I20240503 14:12:09.545410 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7fdf02629122402c9cbf8053f89eced3" } modify_external_catalogs: true
I20240503 14:12:09.545980 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a0b97d2f3edd4c549bbd9c1748f4d82f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.546747 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 1751f775fe6943d9b5fcc3b113393a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.546728 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_63c1e1044be8466cb29bf56927fb925e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.547392 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.547417 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_aa516ed89b714cf788cdf47928d6a415" } new_table_name: "default.Table_30c516ddaa234806bb557849ada87562" modify_external_catalogs: true
I20240503 14:12:09.547693 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.550504 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 40da5f5fff94462d8af740f3ee337e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.551364 28661 ts_tablet_manager.cc:1875] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:09.553259 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2331aec871824a238f3f64de217d7092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2331aec871824a238f3f64de217d7092 already in progress: creating tablet
W20240503 14:12:09.553390 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0b97d2f3edd4c549bbd9c1748f4d82f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.553522 28604 tablet_bootstrap.cc:654] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:09.553306 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f5415973b9641a3a612e9236aad1e92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:09.556394 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63c1e1044be8466cb29bf56927fb925e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.558048 28653 ts_tablet_manager.cc:1888] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.560343 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d2600031df4d77b9afe636a9b5ad40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.560555 28604 tablet_bootstrap.cc:492] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.560727 28604 ts_tablet_manager.cc:1367] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240503 14:12:09.565317 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 495190fca06a40a386c8215eb2195920 with delete_type TABLET_DATA_DELETED (replaced by tablet 7261088abbc641db8d737a48bbac25bd) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.565640 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.565886 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet d93fea5caec94695a5df26d3630c50c7 with delete_type TABLET_DATA_DELETED (replaced by tablet b476e87faea1415d95ae014a93fa22af) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.568186 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cd09e769be4bee95bbef9c3c550451 with delete_type TABLET_DATA_DELETED (replaced by tablet 49445f0f1dd1451c9ba066e61273f2f1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.568250 28604 raft_consensus.cc:348] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.568511 28604 raft_consensus.cc:374] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.568603 28604 raft_consensus.cc:729] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.568684 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.559154 28600 tablet_bootstrap.cc:492] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.568846 28661 ts_tablet_manager.cc:1888] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.568943 28600 ts_tablet_manager.cc:1367] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.007s
I20240503 14:12:09.568989 28661 log.cc:1196] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/59a91f22268346b38ae35f96b0d763f7
I20240503 14:12:09.569383 28661 ts_tablet_manager.cc:1909] T 59a91f22268346b38ae35f96b0d763f7 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.569931 28600 raft_consensus.cc:348] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.570245 28600 raft_consensus.cc:374] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.570531 29243 raft_consensus.cc:482] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.570758 29243 raft_consensus.cc:504] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.571470 29243 leader_election.cc:288] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.572906 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:09.576553 28150 leader_election.cc:341] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:09.576869 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed4dc732bcb44222b4b15e5d77defe33 consensus state has no leader
I20240503 14:12:09.577116 28614 tablet_bootstrap.cc:492] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.577319 28614 ts_tablet_manager.cc:1367] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.155s user 0.031s sys 0.021s
W20240503 14:12:09.577371 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9011de10a7ff497fb35ccb3ada7315ef consensus state has no leader
W20240503 14:12:09.578596 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a703cfbaad2a4a618abb9e07b4930cee consensus state has no leader
I20240503 14:12:09.579802 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.578151 28614 raft_consensus.cc:348] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.580039 28614 raft_consensus.cc:374] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.580161 28614 raft_consensus.cc:729] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.580972 28614 consensus_queue.cc:260] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.581346 28661 tablet_replica.cc:330] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.581569 28661 raft_consensus.cc:2229] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:09.581712 28150 leader_election.cc:341] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.581887 28150 leader_election.cc:302] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.582491 29243 raft_consensus.cc:2726] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.586431 28600 raft_consensus.cc:729] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.586915 28600 consensus_queue.cc:260] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.587177 28661 pending_rounds.cc:62] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:09.587267 28661 pending_rounds.cc:68] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597687889440768 op_type: NO_OP noop_request { }
I20240503 14:12:09.587417 28661 raft_consensus.cc:2866] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:09.587533 28661 raft_consensus.cc:2258] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.589746 28600 ts_tablet_manager.cc:1398] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240503 14:12:09.590440 28600 tablet_bootstrap.cc:492] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.591866 28661 ts_tablet_manager.cc:1875] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.592495 28600 tablet_bootstrap.cc:654] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.593631 28653 log.cc:1196] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/74df7781cc6a4dd6b6fd78a728ffd71b
I20240503 14:12:09.594192 28653 ts_tablet_manager.cc:1909] T 74df7781cc6a4dd6b6fd78a728ffd71b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:09.594481 28604 consensus_queue.cc:260] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.596171 28604 ts_tablet_manager.cc:1398] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20240503 14:12:09.596048 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_285af15c5e1c49808d97cbcacedbc830"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.596720 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_285af15c5e1c49808d97cbcacedbc830 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.598873 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c2d18c6af36b4d0096425aa0afc52fd2" } new_table_name: "default.Table_f2493099c9fb43b7b2f070935a6a5f1f" modify_external_catalogs: true
I20240503 14:12:09.602600 28653 tablet_replica.cc:330] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.602833 28604 tablet_bootstrap.cc:492] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.603474 28653 raft_consensus.cc:2229] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.604333 28653 raft_consensus.cc:2258] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.605370 28653 ts_tablet_manager.cc:1875] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.606561 28604 tablet_bootstrap.cc:654] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.607802 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 74df7781cc6a4dd6b6fd78a728ffd71b (table default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]) successfully deleted
W20240503 14:12:09.608175 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 495190fca06a40a386c8215eb2195920 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 495190fca06a40a386c8215eb2195920 already in progress: creating tablet
W20240503 14:12:09.608445 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 8fe3affd0d144b1fb0078e7690b9b36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fe3affd0d144b1fb0078e7690b9b36d already in progress: creating tablet
W20240503 14:12:09.608683 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d93fea5caec94695a5df26d3630c50c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d93fea5caec94695a5df26d3630c50c7 already in progress: creating tablet
W20240503 14:12:09.608893 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 80cd09e769be4bee95bbef9c3c550451 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80cd09e769be4bee95bbef9c3c550451 already in progress: creating tablet
W20240503 14:12:09.609148 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b2bba6a3d6e84552b027f23a29cbb38a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2bba6a3d6e84552b027f23a29cbb38a already in progress: creating tablet
I20240503 14:12:09.609197 28614 ts_tablet_manager.cc:1398] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20240503 14:12:09.609685 28614 tablet_bootstrap.cc:492] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.610739 28661 ts_tablet_manager.cc:1888] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.610898 28661 log.cc:1196] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/779f373ef12a4da091309fcf247b5693
I20240503 14:12:09.611299 28661 ts_tablet_manager.cc:1909] T 779f373ef12a4da091309fcf247b5693 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.611325 28600 tablet_bootstrap.cc:492] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.611553 28600 ts_tablet_manager.cc:1367] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 14:12:09.611574 28614 tablet_bootstrap.cc:654] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.612448 28600 raft_consensus.cc:348] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.612762 28600 raft_consensus.cc:374] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.612870 28600 raft_consensus.cc:729] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.613173 28600 consensus_queue.cc:260] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:09.609366 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 80cd09e769be4bee95bbef9c3c550451 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80cd09e769be4bee95bbef9c3c550451 already in progress: creating tablet
I20240503 14:12:09.615406 28600 ts_tablet_manager.cc:1398] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:09.616366 28600 tablet_bootstrap.cc:492] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.618279 28600 tablet_bootstrap.cc:654] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.620026 28661 tablet_replica.cc:330] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.620294 28661 raft_consensus.cc:2229] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.620564 28661 raft_consensus.cc:2258] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:09.621260 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b2bba6a3d6e84552b027f23a29cbb38a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2bba6a3d6e84552b027f23a29cbb38a already in progress: creating tablet
I20240503 14:12:09.621378 28661 ts_tablet_manager.cc:1875] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.621541 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 59a91f22268346b38ae35f96b0d763f7 (table default.Table_59af1c61c0534502b0f2cdaff98f4e96 [id=434814ed785f43e380e9c8b1cd884cf2]) successfully deleted
I20240503 14:12:09.621749 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 779f373ef12a4da091309fcf247b5693 (table default.Table_8c90a0cf9e1f495ea0ea85f1f8726f52 [id=857b2f49969d4291b98de4f3776e627d]) successfully deleted
W20240503 14:12:09.621940 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1751f775fe6943d9b5fcc3b113393a73 because the tablet was not found. No further retry: Not found: Tablet not found: 1751f775fe6943d9b5fcc3b113393a73
I20240503 14:12:09.624955 28653 ts_tablet_manager.cc:1888] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.624855 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4eec071da1eb4980b74b08ce5b21765b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.625535 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4eec071da1eb4980b74b08ce5b21765b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.627593 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4515f7561eae476abe1e2f9b2cf55b1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.628201 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4515f7561eae476abe1e2f9b2cf55b1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.629835 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2a170feff0884500b519684ac43a06b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.630424 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a170feff0884500b519684ac43a06b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.632226 28028 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]). Expected version 1 got 0
I20240503 14:12:09.632580 28028 catalog_manager.cc:5484] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:09.633066 28028 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet e9609112657f427cbeeb3ef8f28cddf0 (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]). Expected version 5 got 0
I20240503 14:12:09.633421 28604 tablet_bootstrap.cc:492] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.633580 28604 ts_tablet_manager.cc:1367] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240503 14:12:09.634409 28604 raft_consensus.cc:348] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.634629 28604 raft_consensus.cc:374] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.634722 28604 raft_consensus.cc:729] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.634968 28604 consensus_queue.cc:260] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.635560 28604 ts_tablet_manager.cc:1398] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:09.636986 28604 tablet_bootstrap.cc:492] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.639016 28653 log.cc:1196] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/40da5f5fff94462d8af740f3ee337e76
I20240503 14:12:09.639101 28661 ts_tablet_manager.cc:1888] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.639214 28661 log.cc:1196] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/75dafd98b05d4c63bbf30165a2ac8d1e
I20240503 14:12:09.639430 28653 ts_tablet_manager.cc:1909] T 40da5f5fff94462d8af740f3ee337e76 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:09.639428 29266 raft_consensus.cc:482] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.639556 28661 ts_tablet_manager.cc:1909] T 75dafd98b05d4c63bbf30165a2ac8d1e P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:09.639984 28032 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_ea4ed2f332dc46aaa1f6e7025a1a1c3c [id=5cf73f88a73a4cc5966467ecfff2140f] alter complete (version 2)
I20240503 14:12:09.639999 28328 tablet_service.cc:1467] Processing CreateTablet for tablet ce4846d662a64d51b16fd5447c371e9a (DEFAULT_TABLE table=default.Table_4fdbe31ba57a42b993d60fa19081c044 [id=ea4ef9402edd4db9820a6a318bfec37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.640467 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce4846d662a64d51b16fd5447c371e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.629293 28600 tablet_bootstrap.cc:492] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.641507 28600 ts_tablet_manager.cc:1367] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20240503 14:12:09.642375 28600 raft_consensus.cc:348] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.642596 28600 raft_consensus.cc:374] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.642675 28600 raft_consensus.cc:729] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.642930 28600 consensus_queue.cc:260] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.644587 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_922e97aa11034bbda5a51b53fc131880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.645190 28600 ts_tablet_manager.cc:1398] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20240503 14:12:09.645185 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_922e97aa11034bbda5a51b53fc131880 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.645483 28600 tablet_bootstrap.cc:492] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.647105 28600 tablet_bootstrap.cc:654] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.649104 28604 tablet_bootstrap.cc:654] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.649770 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet da19787034c24ebe929601d49cb21b3b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:09.651767 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ff5facc5efa54652826a5f626e9aca85" } new_table_name: "default.Table_472814febc0548599d68e796d57f48a1" modify_external_catalogs: true
I20240503 14:12:09.639648 29266 raft_consensus.cc:504] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.652928 28661 tablet_replica.cc:330] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:09.653051 29266 leader_election.cc:288] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.653162 28661 raft_consensus.cc:2229] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.653329 28661 raft_consensus.cc:2258] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.654923 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5992e28f1745678618230c183609e4" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.657142 28195 tablet_service.cc:1467] Processing CreateTablet for tablet ce4846d662a64d51b16fd5447c371e9a (DEFAULT_TABLE table=default.Table_4fdbe31ba57a42b993d60fa19081c044 [id=ea4ef9402edd4db9820a6a318bfec37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.657603 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce4846d662a64d51b16fd5447c371e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.657963 29265 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:09.658238 29265 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:09.659400 28328 tablet_service.cc:1467] Processing CreateTablet for tablet a3d1314841574fc8bcf3ccb16755e421 (DEFAULT_TABLE table=default.Table_a1bf5785458b47d0a18f3f3088449ea3 [id=e5c443cd73d340e99806c89d5b8e659f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.659904 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d1314841574fc8bcf3ccb16755e421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.660293 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5992e28f1745678618230c183609e4" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:09.660589 28212 raft_consensus.cc:2379] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:09.661176 28474 tablet_service.cc:1467] Processing CreateTablet for tablet ce4846d662a64d51b16fd5447c371e9a (DEFAULT_TABLE table=default.Table_4fdbe31ba57a42b993d60fa19081c044 [id=ea4ef9402edd4db9820a6a318bfec37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.661659 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce4846d662a64d51b16fd5447c371e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.663280 28604 tablet_bootstrap.cc:492] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:09.663499 28604 ts_tablet_manager.cc:1367] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240503 14:12:09.664445 28604 raft_consensus.cc:348] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.664673 28604 raft_consensus.cc:374] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.664738 28604 raft_consensus.cc:729] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.664968 28290 leader_election.cc:302] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.664995 28604 consensus_queue.cc:260] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.665292 29266 raft_consensus.cc:3037] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.665453 29180 raft_consensus.cc:482] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.665629 29180 raft_consensus.cc:504] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.666360 29180 leader_election.cc:288] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.667137 28604 ts_tablet_manager.cc:1398] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:09.667495 28604 tablet_bootstrap.cc:492] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:09.668205 29266 raft_consensus.cc:2726] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.668252 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989ea2f962a7492d8c32e4becec1457d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:09.668496 28212 raft_consensus.cc:2454] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:09.668969 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989ea2f962a7492d8c32e4becec1457d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.669219 28352 raft_consensus.cc:2454] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:09.669487 28427 leader_election.cc:302] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:09.669742 28179 tablet_service.cc:1467] Processing CreateTablet for tablet a3d1314841574fc8bcf3ccb16755e421 (DEFAULT_TABLE table=default.Table_a1bf5785458b47d0a18f3f3088449ea3 [id=e5c443cd73d340e99806c89d5b8e659f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.670212 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d1314841574fc8bcf3ccb16755e421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.670459 28472 tablet_service.cc:1467] Processing CreateTablet for tablet a3d1314841574fc8bcf3ccb16755e421 (DEFAULT_TABLE table=default.Table_a1bf5785458b47d0a18f3f3088449ea3 [id=e5c443cd73d340e99806c89d5b8e659f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.670915 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d1314841574fc8bcf3ccb16755e421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.671154 28604 tablet_bootstrap.cc:654] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.673005 28600 tablet_bootstrap.cc:492] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.673204 28600 ts_tablet_manager.cc:1367] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20240503 14:12:09.674070 28600 raft_consensus.cc:348] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.674326 28600 raft_consensus.cc:374] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.674399 28600 raft_consensus.cc:729] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.674702 28600 consensus_queue.cc:260] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.676266 29182 raft_consensus.cc:2781] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.676383 29182 raft_consensus.cc:482] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.676482 29182 raft_consensus.cc:3037] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.677596 28600 ts_tablet_manager.cc:1398] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:09.677722 28614 tablet_bootstrap.cc:492] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.677886 28614 ts_tablet_manager.cc:1367] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.008s
I20240503 14:12:09.678202 28600 tablet_bootstrap.cc:492] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.678711 28614 raft_consensus.cc:348] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.678944 28614 raft_consensus.cc:374] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.679044 28614 raft_consensus.cc:729] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.679026 29182 raft_consensus.cc:504] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.679278 28614 consensus_queue.cc:260] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.679746 29182 leader_election.cc:288] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.679874 28614 ts_tablet_manager.cc:1398] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:09.679944 28600 tablet_bootstrap.cc:654] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.680338 28614 tablet_bootstrap.cc:492] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.681332 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989ea2f962a7492d8c32e4becec1457d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:09.681535 28212 raft_consensus.cc:3037] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.682158 28614 tablet_bootstrap.cc:654] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.685061 28212 raft_consensus.cc:2454] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:09.685755 28427 leader_election.cc:302] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:09.689392 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989ea2f962a7492d8c32e4becec1457d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.689652 28352 raft_consensus.cc:3037] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.689996 29182 raft_consensus.cc:2781] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.690460 29182 raft_consensus.cc:686] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:09.690784 29182 consensus_queue.cc:237] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.692792 28352 raft_consensus.cc:2454] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:09.697350 28600 tablet_bootstrap.cc:492] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.697644 28600 ts_tablet_manager.cc:1367] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240503 14:12:09.698664 28600 raft_consensus.cc:348] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.699431 28474 tablet_service.cc:1467] Processing CreateTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 (DEFAULT_TABLE table=default.Table_b86f741291e64aef9908c5e036e4cce8 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.699997 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c816f3b05cfb4496984f6cb28d310cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.700410 28600 raft_consensus.cc:374] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.700652 28661 ts_tablet_manager.cc:1875] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.704234 28600 raft_consensus.cc:729] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.704989 28600 consensus_queue.cc:260] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.706041 28195 tablet_service.cc:1467] Processing CreateTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 (DEFAULT_TABLE table=default.Table_b86f741291e64aef9908c5e036e4cce8 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.707387 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c816f3b05cfb4496984f6cb28d310cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.709362 28600 ts_tablet_manager.cc:1398] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 14:12:09.712141 28334 tablet_service.cc:1467] Processing CreateTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 (DEFAULT_TABLE table=default.Table_b86f741291e64aef9908c5e036e4cce8 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.712726 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c816f3b05cfb4496984f6cb28d310cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.713538 28600 tablet_bootstrap.cc:492] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.715322 28600 tablet_bootstrap.cc:654] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.719556 29243 raft_consensus.cc:482] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.719811 29243 raft_consensus.cc:504] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.720589 29243 leader_election.cc:288] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.721906 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd5234791524eddb7edd493e39fb993" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.723016 29266 raft_consensus.cc:482] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.723343 29266 raft_consensus.cc:504] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:09.723590 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd5234791524eddb7edd493e39fb993" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.724188 29266 leader_election.cc:288] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.724809 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 780b53efefcf42569bba2fcabd21e211 (DEFAULT_TABLE table=default.Table_98de9a28340945fc977e41d46d3361c0 [id=e3a00fad0fa8492faf33675df3d18f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.725483 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780b53efefcf42569bba2fcabd21e211. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:09.728379 28150 leader_election.cc:341] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:09.728672 28150 leader_election.cc:341] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.728632 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 780b53efefcf42569bba2fcabd21e211 (DEFAULT_TABLE table=default.Table_98de9a28340945fc977e41d46d3361c0 [id=e3a00fad0fa8492faf33675df3d18f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.728818 28150 leader_election.cc:302] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.729182 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780b53efefcf42569bba2fcabd21e211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.729684 29243 raft_consensus.cc:2726] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.735858 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 920b41e3b21347b393497de4d4e87469 (DEFAULT_TABLE table=default.Table_28c16c4a952d45c0b34fd1d4864ac748 [id=2fd597e69c1041a2984ca535058f6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.736481 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920b41e3b21347b393497de4d4e87469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.745230 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 920b41e3b21347b393497de4d4e87469 (DEFAULT_TABLE table=default.Table_28c16c4a952d45c0b34fd1d4864ac748 [id=2fd597e69c1041a2984ca535058f6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.745720 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920b41e3b21347b393497de4d4e87469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.753836 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 780b53efefcf42569bba2fcabd21e211 (DEFAULT_TABLE table=default.Table_98de9a28340945fc977e41d46d3361c0 [id=e3a00fad0fa8492faf33675df3d18f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.754344 29180 consensus_queue.cc:1035] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:09.754530 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780b53efefcf42569bba2fcabd21e211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.759526 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 (DEFAULT_TABLE table=default.Table_3169e805aa1543889ee0dab4758ae9e5 [id=9e438ac1c3254c7aae6f81c707f433d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.760107 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cad43f3c6094253ba87d0ed3fc29fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.760403 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 920b41e3b21347b393497de4d4e87469 (DEFAULT_TABLE table=default.Table_28c16c4a952d45c0b34fd1d4864ac748 [id=2fd597e69c1041a2984ca535058f6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.760813 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920b41e3b21347b393497de4d4e87469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.761188 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 40da5f5fff94462d8af740f3ee337e76 (table default.Table_63619685889a4263ae065b254160cf30 [id=6a81b734738e4d998215938d5ca5dad7]) successfully deleted
W20240503 14:12:09.761489 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2331aec871824a238f3f64de217d7092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2331aec871824a238f3f64de217d7092 already in progress: creating tablet
W20240503 14:12:09.761703 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 495190fca06a40a386c8215eb2195920 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 495190fca06a40a386c8215eb2195920 already in progress: creating tablet
W20240503 14:12:09.761881 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 8fe3affd0d144b1fb0078e7690b9b36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fe3affd0d144b1fb0078e7690b9b36d already in progress: creating tablet
W20240503 14:12:09.762935 28290 leader_election.cc:341] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 1693dd36a3d84d88a49690c537590083
W20240503 14:12:09.763245 28290 leader_election.cc:341] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 1693dd36a3d84d88a49690c537590083
I20240503 14:12:09.763361 28290 leader_election.cc:302] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.763628 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 86d810ab0cf34941a343eef1593632bf (DEFAULT_TABLE table=default.Table_a1315a4233714b3f81e5a926c45e14c7 [id=c358e28088d44cf598c96a1bd2a4321e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.763741 29266 raft_consensus.cc:2726] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.764040 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d810ab0cf34941a343eef1593632bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.764405 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 02d45788ae644a8581acca54cc55dd3f (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.764925 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d45788ae644a8581acca54cc55dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.765008 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 (DEFAULT_TABLE table=default.Table_3169e805aa1543889ee0dab4758ae9e5 [id=9e438ac1c3254c7aae6f81c707f433d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.765399 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cad43f3c6094253ba87d0ed3fc29fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.766166 28182 tablet_service.cc:1467] Processing CreateTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.766546 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e4f6e3908e420e9fa156ee24b437d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.773598 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 (DEFAULT_TABLE table=default.Table_ff5facc5efa54652826a5f626e9aca85 [id=7128b9ce63544947ab63aa6d2490c691]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.773824 29243 raft_consensus.cc:482] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.774127 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac2833a0bb045dcb2824468d936ffe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.774022 29243 raft_consensus.cc:504] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.774806 29243 leader_election.cc:288] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.776024 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc230b489ab8455c8cb2eba31084fddb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.776216 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df (DEFAULT_TABLE table=default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.776808 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc230b489ab8455c8cb2eba31084fddb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.777068 28489 raft_consensus.cc:2454] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:09.778383 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7b2865e9604ce58280321c7ee2a88e (DEFAULT_TABLE table=default.Table_38aab413057643f78d942ab06e5905f7 [id=32be054076954aed9265888ca62e510b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.778743 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 11e14e8b5f4046029af4673a277112d9 (DEFAULT_TABLE table=default.Table_68199a1d18244f54bb0b7b0256a58e2a [id=2528ba5e3fe74c40949dcc7c289d0630]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.779901 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 02d45788ae644a8581acca54cc55dd3f (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.780372 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d45788ae644a8581acca54cc55dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.781049 29180 consensus_queue.cc:1035] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:09.781373 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.782837 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 02d45788ae644a8581acca54cc55dd3f (DEFAULT_TABLE table=default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.783258 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d45788ae644a8581acca54cc55dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.787294 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df (DEFAULT_TABLE table=default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.787739 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.788132 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e14e8b5f4046029af4673a277112d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.788873 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 11e14e8b5f4046029af4673a277112d9 (DEFAULT_TABLE table=default.Table_68199a1d18244f54bb0b7b0256a58e2a [id=2528ba5e3fe74c40949dcc7c289d0630]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.789348 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e14e8b5f4046029af4673a277112d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.789711 28150 leader_election.cc:302] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:09.790117 28150 leader_election.cc:341] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.790621 29113 raft_consensus.cc:2781] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.790810 29113 raft_consensus.cc:482] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.790908 29113 raft_consensus.cc:3037] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.795080 29180 raft_consensus.cc:482] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.795274 29180 raft_consensus.cc:504] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.796000 29180 leader_election.cc:288] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:09.797593 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 11e14e8b5f4046029af4673a277112d9 (DEFAULT_TABLE table=default.Table_68199a1d18244f54bb0b7b0256a58e2a [id=2528ba5e3fe74c40949dcc7c289d0630]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.797722 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 86d810ab0cf34941a343eef1593632bf (DEFAULT_TABLE table=default.Table_a1315a4233714b3f81e5a926c45e14c7 [id=c358e28088d44cf598c96a1bd2a4321e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.798189 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e14e8b5f4046029af4673a277112d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.798249 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d810ab0cf34941a343eef1593632bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.798909 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7b2865e9604ce58280321c7ee2a88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.807540 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.808293 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e4f6e3908e420e9fa156ee24b437d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.809090 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 779f373ef12a4da091309fcf247b5693 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.809509 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dab0296a0894a63afc16ed466c591d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.809847 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8411f474c247c2abfd0bd9682fb8a2) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.810173 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet f50ad403bd354559b2522a76af1fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:09.811350 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df (DEFAULT_TABLE table=default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.811808 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.816241 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 (DEFAULT_TABLE table=default.Table_ff5facc5efa54652826a5f626e9aca85 [id=7128b9ce63544947ab63aa6d2490c691]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.816300 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7b2865e9604ce58280321c7ee2a88e (DEFAULT_TABLE table=default.Table_38aab413057643f78d942ab06e5905f7 [id=32be054076954aed9265888ca62e510b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.816726 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac2833a0bb045dcb2824468d936ffe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.816984 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7b2865e9604ce58280321c7ee2a88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.822587 29243 raft_consensus.cc:482] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.822824 29243 raft_consensus.cc:504] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.829193 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32b6995818041778617a0d6bf936c8c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.829631 29113 raft_consensus.cc:504] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:09.830191 28150 leader_election.cc:341] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.830448 29113 leader_election.cc:288] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.831328 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc230b489ab8455c8cb2eba31084fddb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:09.831842 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc230b489ab8455c8cb2eba31084fddb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:09.832068 28489 raft_consensus.cc:3037] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.835846 28489 raft_consensus.cc:2454] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:09.836483 28150 leader_election.cc:302] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:09.836728 29243 leader_election.cc:288] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.836936 29243 raft_consensus.cc:2781] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.837102 29243 raft_consensus.cc:686] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:09.837396 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32b6995818041778617a0d6bf936c8c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:09.837409 29243 consensus_queue.cc:237] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:09.838001 28150 leader_election.cc:341] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.838171 28150 leader_election.cc:302] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.838675 29113 raft_consensus.cc:2726] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.845804 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 (DEFAULT_TABLE table=default.Table_ff5facc5efa54652826a5f626e9aca85 [id=7128b9ce63544947ab63aa6d2490c691]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.846513 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac2833a0bb045dcb2824468d936ffe4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:09.846632 28427 leader_election.cc:341] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6dab0296a0894a63afc16ed466c591d5
I20240503 14:12:09.847224 29182 consensus_queue.cc:1035] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:09.849678 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d93fea5caec94695a5df26d3630c50c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d93fea5caec94695a5df26d3630c50c7 already in progress: creating tablet
W20240503 14:12:09.849910 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 87856.3us
W20240503 14:12:09.850262 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 80cd09e769be4bee95bbef9c3c550451 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80cd09e769be4bee95bbef9c3c550451 already in progress: creating tablet
W20240503 14:12:09.850500 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b2bba6a3d6e84552b027f23a29cbb38a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2bba6a3d6e84552b027f23a29cbb38a already in progress: creating tablet
W20240503 14:12:09.855558 28150 leader_election.cc:341] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.851140 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 86d810ab0cf34941a343eef1593632bf (DEFAULT_TABLE table=default.Table_a1315a4233714b3f81e5a926c45e14c7 [id=c358e28088d44cf598c96a1bd2a4321e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:09.872987 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet cae4969fc9474fdbaeae1b4d09e4875b because the tablet was not found. No further retry: Not found: Tablet not found: cae4969fc9474fdbaeae1b4d09e4875b
I20240503 14:12:09.874601 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 75dafd98b05d4c63bbf30165a2ac8d1e (table default.Table_986d778bcd814c9c87dbd18d1f485cc9 [id=5963eb580afa4dc18cf4e472ed1fb76c]) successfully deleted
I20240503 14:12:09.875531 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.882870 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 351d5bfb12c34c5f9de90e2df1b36fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet d6f593467bf4482cb926b63480790ad4) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.883244 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a91f22268346b38ae35f96b0d763f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.883491 28603 tablet_replica.cc:330] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:09.884250 28603 raft_consensus.cc:2229] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.884488 28603 raft_consensus.cc:2258] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.885198 28603 ts_tablet_manager.cc:1875] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.886055 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 779f373ef12a4da091309fcf247b5693 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.888460 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d810ab0cf34941a343eef1593632bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.901158 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet f51834695b8b4a729a088da81eee2e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.901808 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 1693dd36a3d84d88a49690c537590083 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7c5dfac02d4dfe932996d2a37fe031) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.904786 28603 ts_tablet_manager.cc:1888] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.907750 28603 log.cc:1196] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/351d5bfb12c34c5f9de90e2df1b36fc1
I20240503 14:12:09.916360 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dab0296a0894a63afc16ed466c591d5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.916565 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 (DEFAULT_TABLE table=default.Table_3169e805aa1543889ee0dab4758ae9e5 [id=9e438ac1c3254c7aae6f81c707f433d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.917232 28603 ts_tablet_manager.cc:1909] T 351d5bfb12c34c5f9de90e2df1b36fc1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:09.917368 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e4f6e3908e420e9fa156ee24b437d6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:09.922680 28427 leader_election.cc:341] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.922858 28427 leader_election.cc:302] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:09.920583 29266 raft_consensus.cc:482] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.926815 29266 raft_consensus.cc:504] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.927249 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cad43f3c6094253ba87d0ed3fc29fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.927556 28600 tablet_bootstrap.cc:492] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.927604 29266 leader_election.cc:288] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.927750 28600 ts_tablet_manager.cc:1367] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.214s user 0.028s sys 0.050s
I20240503 14:12:09.923265 28357 raft_consensus.cc:3037] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.924153 28614 tablet_bootstrap.cc:492] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:09.928170 28614 ts_tablet_manager.cc:1367] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.248s user 0.062s sys 0.089s
I20240503 14:12:09.929036 28614 raft_consensus.cc:348] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.929279 28614 raft_consensus.cc:374] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.929351 28614 raft_consensus.cc:729] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.929229 28600 raft_consensus.cc:348] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.929468 28600 raft_consensus.cc:374] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.929553 28600 raft_consensus.cc:729] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.929605 28614 consensus_queue.cc:260] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.930567 28614 ts_tablet_manager.cc:1398] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:09.930521 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7b2865e9604ce58280321c7ee2a88e (DEFAULT_TABLE table=default.Table_38aab413057643f78d942ab06e5905f7 [id=32be054076954aed9265888ca62e510b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.931102 28600 consensus_queue.cc:260] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.931874 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7b2865e9604ce58280321c7ee2a88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.933072 28600 ts_tablet_manager.cc:1398] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20240503 14:12:09.933229 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc230b489ab8455c8cb2eba31084fddb consensus state has no leader
I20240503 14:12:09.933424 28600 tablet_bootstrap.cc:492] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.933593 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 1603e2360041427f8635782e812b6431 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:09.934255 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_f2493099c9fb43b7b2f070935a6a5f1f [id=084b186eba154ce1939f530379f4ce76]). Expected version 1 got 0
I20240503 14:12:09.935166 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4d256c29f9b7400497caee67285a9831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.935757 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d256c29f9b7400497caee67285a9831 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.936430 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0b139ab0b5864024ac48215d46b0a7bd
I20240503 14:12:09.936611 28600 tablet_bootstrap.cc:654] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.936584 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_627b41e76476426da74d6722301bc59f" } modify_external_catalogs: true
I20240503 14:12:09.936775 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_627b41e76476426da74d6722301bc59f" } modify_external_catalogs: true
I20240503 14:12:09.936959 28024 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 421) took 1078 ms. Trace:
I20240503 14:12:09.937070 28024 rpcz_store.cc:276] 0503 14:12:08.858795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.483609 (+624814us) service_pool.cc:225] Handling call
0503 14:12:09.484131 (+ 522us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:09.484284 (+ 153us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:09.484361 (+ 77us) catalog_manager.cc:2758] Locking tablets
0503 14:12:09.631521 (+147160us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:09.936209 (+304688us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:09.936294 (+ 85us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:09.936335 (+ 41us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:09.936918 (+ 583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:09.632354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:09.632451 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:09.632456 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:09.632458 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:09.632564 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:09.632577 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:09.632581 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:09.632701 (+ 120us) tablet.cc:699] Row locks acquired
0503 14:12:09.632704 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:09.632739 (+ 35us) write_op.cc:270] Start()
0503 14:12:09.632768 (+ 29us) write_op.cc:276] Timestamp: P: 1714745529632725 usec, L: 0
0503 14:12:09.632771 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:09.632926 (+ 155us) log.cc:842] Serialized 762 byte log entry
0503 14:12:09.929609 (+296683us) op_driver.cc:444] REPLICATION: finished
0503 14:12:09.931852 (+ 2243us) write_op.cc:301] APPLY: starting
0503 14:12:09.931862 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:09.931891 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:09.931911 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:09.931963 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:09.931987 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:09.931994 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:09.932042 (+ 48us) log.cc:842] Serialized 29 byte log entry
0503 14:12:09.935943 (+ 3901us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:09.935974 (+ 31us) write_op.cc:454] Released schema lock
0503 14:12:09.935983 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:09.936052 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:09.936060 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2214,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":4291,"num_ops":2,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":657,"prepare.run_wall_time_us":657,"raft.queue_time_us":120,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":296824}]]}
I20240503 14:12:09.937821 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc77503fb1ad4d03a6278aef9f3ab979" } modify_external_catalogs: true
I20240503 14:12:09.938006 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc77503fb1ad4d03a6278aef9f3ab979" } modify_external_catalogs: true
I20240503 14:12:09.939123 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_43c9760cf3b54d68b218cafe8393cda2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.941494 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 65503.3us
I20240503 14:12:09.943127 28600 tablet_bootstrap.cc:492] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:09.943327 28600 ts_tablet_manager.cc:1367] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:12:09.942459 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e0175b38c1e0464a9a6208cfaf68ca0d
I20240503 14:12:09.944041 28030 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 428) took 1021 ms. Trace:
I20240503 14:12:09.944228 28030 rpcz_store.cc:276] 0503 14:12:08.922877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.524881 (+602004us) service_pool.cc:225] Handling call
0503 14:12:09.545136 (+ 20255us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedfca6 00007fe123ed1f66 00007fe123ed1c95 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:09.545507 (+ 371us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:09.545645 (+ 138us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:09.545702 (+ 57us) catalog_manager.cc:2758] Locking tablets
0503 14:12:09.633804 (+ 88102us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:09.942276 (+308472us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:09.942364 (+ 88us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:09.942410 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:09.943974 (+ 1564us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:09.634425 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:09.634511 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:09.634516 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:09.634518 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:09.634609 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:09.634618 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:09.634622 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:09.634735 (+ 113us) tablet.cc:699] Row locks acquired
0503 14:12:09.634738 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:09.634769 (+ 31us) write_op.cc:270] Start()
0503 14:12:09.634799 (+ 30us) write_op.cc:276] Timestamp: P: 1714745529634756 usec, L: 0
0503 14:12:09.634803 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:09.634938 (+ 135us) log.cc:842] Serialized 731 byte log entry
0503 14:12:09.929673 (+294735us) op_driver.cc:444] REPLICATION: finished
0503 14:12:09.932532 (+ 2859us) write_op.cc:301] APPLY: starting
0503 14:12:09.932542 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:09.932585 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:09.932605 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:09.932674 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:09.932698 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:09.932704 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:09.932753 (+ 49us) log.cc:842] Serialized 29 byte log entry
0503 14:12:09.932789 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:09.932802 (+ 13us) write_op.cc:454] Released schema lock
0503 14:12:09.932807 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:09.932854 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:09.932857 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":45896064,"child_traces":[["op",{"apply.queue_time_us":2830,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":366,"num_ops":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":591,"prepare.run_wall_time_us":590,"raft.queue_time_us":102,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":294859}]]}
I20240503 14:12:09.945089 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_94abbca6df6f456fb210dd71f21384af" } modify_external_catalogs: true
I20240503 14:12:09.945290 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_94abbca6df6f456fb210dd71f21384af" } modify_external_catalogs: true
I20240503 14:12:09.945825 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:09.946638 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43c9760cf3b54d68b218cafe8393cda2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:09.946625 28290 leader_election.cc:341] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: f51834695b8b4a729a088da81eee2e47
W20240503 14:12:09.946646 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e0175b38c1e0464a9a6208cfaf68ca0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0175b38c1e0464a9a6208cfaf68ca0d already in progress: creating tablet
W20240503 14:12:09.946872 28290 leader_election.cc:341] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: f51834695b8b4a729a088da81eee2e47
I20240503 14:12:09.946969 28290 leader_election.cc:302] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:09.948479 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.948774 29266 raft_consensus.cc:2726] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:09.952679 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.953168 28614 tablet_bootstrap.cc:492] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:09.954682 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.955260 28614 tablet_bootstrap.cc:654] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.956429 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 2f89fca0480d4bf28160384ae0421bce from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:09.957108 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_701e9c2bdea449d49f86dffb86090877" } new_table_name: "default.Table_2907862ac0af4c3692460a0071e8bce3" modify_external_catalogs: true
I20240503 14:12:09.957720 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1603e2360041427f8635782e812b6431
I20240503 14:12:09.958329 28026 catalog_manager.cc:6554] ReplaceTablet: tablet 1603e2360041427f8635782e812b6431 deleted and replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1
W20240503 14:12:09.958669 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ac2833a0bb045dcb2824468d936ffe4 consensus state has no leader
I20240503 14:12:09.958699 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_285af15c5e1c49808d97cbcacedbc830" } new_table_name: "default.Table_6a18ec9a972b4ad99771cd58f58c1a5b" modify_external_catalogs: true
I20240503 14:12:09.959009 28357 raft_consensus.cc:1261] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:09.959043 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4ec913f6e2544dd98b3a26d8efffcc2f" } new_table_name: "default.Table_730611e6ad48480eb3f4498f3e79725b" modify_external_catalogs: true
I20240503 14:12:09.960357 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet da19787034c24ebe929601d49cb21b3b
I20240503 14:12:09.960727 28603 tablet_replica.cc:330] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:09.960834 28033 catalog_manager.cc:6554] ReplaceTablet: tablet da19787034c24ebe929601d49cb21b3b deleted and replaced by tablet 7d348da459634561985c4ac1710c0bb2
I20240503 14:12:09.960943 28603 raft_consensus.cc:2229] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.961181 28603 raft_consensus.cc:2258] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.961239 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a41a77e2797147c680094d0b130290b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.961786 28603 ts_tablet_manager.cc:1875] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:09.961848 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a41a77e2797147c680094d0b130290b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.963152 29243 raft_consensus.cc:482] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.963347 29243 raft_consensus.cc:504] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:09.963722 28600 raft_consensus.cc:348] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.963922 28600 raft_consensus.cc:374] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.964020 28600 raft_consensus.cc:729] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.964133 29243 leader_election.cc:288] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:09.964315 28600 consensus_queue.cc:260] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:09.964939 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:09.965209 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
W20240503 14:12:09.965706 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0b139ab0b5864024ac48215d46b0a7bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b139ab0b5864024ac48215d46b0a7bd already in progress: creating tablet
I20240503 14:12:09.966470 28600 ts_tablet_manager.cc:1398] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.023s user 0.000s sys 0.003s
I20240503 14:12:09.966830 28600 tablet_bootstrap.cc:492] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:09.967957 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet da19787034c24ebe929601d49cb21b3b with delete_type TABLET_DATA_DELETED (replaced by tablet 7d348da459634561985c4ac1710c0bb2) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.968609 28600 tablet_bootstrap.cc:654] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.969933 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:09.970124 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 351d5bfb12c34c5f9de90e2df1b36fc1 (table default.Table_aa957a5e7a874114aaaec61023360b59 [id=56f6e9c2bcac4a0db2e10e3d68e9f4c1]) successfully deleted
I20240503 14:12:09.970256 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbd3291c56b4c13813c80d6775afe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.973644 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93462d2b1a474bd688d6266cd3b0695c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:09.974090 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 40da5f5fff94462d8af740f3ee337e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:02 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.974463 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dab0296a0894a63afc16ed466c591d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.974525 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 95d9985eecda4f3194c3251216aa58ee
I20240503 14:12:09.974864 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a884cf4ea11e48ef941722b79ad50075
W20240503 14:12:09.975505 28150 leader_election.cc:341] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:09.975404 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7a8f9e31244b4b7aaeb8ea4b458599cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.975572 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_59366cab2245406f967358e760726e9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.976019 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a8f9e31244b4b7aaeb8ea4b458599cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:09.976020 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59366cab2245406f967358e760726e9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.977288 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_848d74c659274863a2c1046f6de2568e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:09.977826 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_848d74c659274863a2c1046f6de2568e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:09.979987 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 1603e2360041427f8635782e812b6431 with delete_type TABLET_DATA_DELETED (replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.980247 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet da19787034c24ebe929601d49cb21b3b with delete_type TABLET_DATA_DELETED (replaced by tablet 7d348da459634561985c4ac1710c0bb2) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.980636 28653 tablet_replica.cc:330] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:09.980861 28653 raft_consensus.cc:2229] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:09.981088 28653 raft_consensus.cc:2258] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:09.981738 28653 ts_tablet_manager.cc:1875] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:09.983383 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d9985eecda4f3194c3251216aa58ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.983958 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.983994 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:09.984187 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d9985eecda4f3194c3251216aa58ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:09.986166 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 0bd5234791524eddb7edd493e39fb993 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:09.993330 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 9ac2833a0bb045dcb2824468d936ffe4 (table default.Table_472814febc0548599d68e796d57f48a1 [id=7128b9ce63544947ab63aa6d2490c691], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9ac2833a0bb045dcb2824468d936ffe4 consensus state has no leader
I20240503 14:12:09.994546 28603 ts_tablet_manager.cc:1888] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:09.996603 28603 log.cc:1196] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/59a91f22268346b38ae35f96b0d763f7
I20240503 14:12:09.996961 28653 ts_tablet_manager.cc:1888] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:09.997164 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 1603e2360041427f8635782e812b6431 with delete_type TABLET_DATA_DELETED (replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:09.997577 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77f0081cc2cb48aaa368741109032fd1 consensus state has no leader
I20240503 14:12:09.997936 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 3d54cc5c03034a6685c1da34f9bd65ac from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:09.998605 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34238c9859724d97a32f2a1d9315edb1 has no consensus state
I20240503 14:12:10.000298 28653 log.cc:1196] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/da19787034c24ebe929601d49cb21b3b
I20240503 14:12:10.000573 28603 ts_tablet_manager.cc:1909] T 59a91f22268346b38ae35f96b0d763f7 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:10.001085 28653 ts_tablet_manager.cc:1909] T da19787034c24ebe929601d49cb21b3b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:10.002363 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet da19787034c24ebe929601d49cb21b3b (table default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
W20240503 14:12:10.003294 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 95d9985eecda4f3194c3251216aa58ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95d9985eecda4f3194c3251216aa58ee already in progress: creating tablet
I20240503 14:12:10.003698 28600 tablet_bootstrap.cc:492] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.003921 28600 ts_tablet_manager.cc:1367] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.003s
W20240503 14:12:10.004040 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a884cf4ea11e48ef941722b79ad50075 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a884cf4ea11e48ef941722b79ad50075 already in progress: creating tablet
W20240503 14:12:10.004752 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 1603e2360041427f8635782e812b6431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1603e2360041427f8635782e812b6431 already in progress: creating tablet
I20240503 14:12:10.004896 28600 raft_consensus.cc:348] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.005040 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 59a91f22268346b38ae35f96b0d763f7 (table default.Table_59af1c61c0534502b0f2cdaff98f4e96 [id=434814ed785f43e380e9c8b1cd884cf2]) successfully deleted
I20240503 14:12:10.005165 28600 raft_consensus.cc:374] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.005260 28600 raft_consensus.cc:729] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.005546 28603 tablet_replica.cc:330] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:10.005542 28600 consensus_queue.cc:260] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.006285 28600 ts_tablet_manager.cc:1398] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.006635 28600 tablet_bootstrap.cc:492] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.006925 28603 raft_consensus.cc:2229] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:10.007441 28603 raft_consensus.cc:2258] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.008436 28600 tablet_bootstrap.cc:654] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.009219 28603 ts_tablet_manager.cc:1875] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:10.016461 29285 log.cc:924] Time spent T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240503 14:12:10.024660 28191 tablet_service.cc:1467] Processing CreateTablet for tablet d832cd30f2414303818d34a0cdf47e60 (DEFAULT_TABLE table=default.Table_a0b97d2f3edd4c549bbd9c1748f4d82f [id=566ce63bc6594fb29df2f8eb42c940e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.025226 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d832cd30f2414303818d34a0cdf47e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.026445 28329 tablet_service.cc:1467] Processing CreateTablet for tablet b59c0bc115d949f69d80e1f3628d5a79 (DEFAULT_TABLE table=default.Table_63c1e1044be8466cb29bf56927fb925e [id=387303a065da4e2ebe6a28ebf492ae46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.027504 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59c0bc115d949f69d80e1f3628d5a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.028473 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 007a892ef8504b848a400c05740eaa0c (DEFAULT_TABLE table=default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.029017 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007a892ef8504b848a400c05740eaa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.029150 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (DEFAULT_TABLE table=default.Table_3f5415973b9641a3a612e9236aad1e92 [id=bd1950deb4274d86bf288d501cf5c710]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.029616 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.031431 28337 tablet_service.cc:1467] Processing CreateTablet for tablet d832cd30f2414303818d34a0cdf47e60 (DEFAULT_TABLE table=default.Table_a0b97d2f3edd4c549bbd9c1748f4d82f [id=566ce63bc6594fb29df2f8eb42c940e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.031966 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d832cd30f2414303818d34a0cdf47e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.032464 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb (DEFAULT_TABLE table=default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a [id=f41fc50b1ca142f290aabb60f6cf0cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.032896 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de9e159d2ba47039dd6bd207a9cddeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.026487 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 007a892ef8504b848a400c05740eaa0c (DEFAULT_TABLE table=default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.035795 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007a892ef8504b848a400c05740eaa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.036506 28180 tablet_service.cc:1467] Processing CreateTablet for tablet b476e87faea1415d95ae014a93fa22af (DEFAULT_TABLE table=default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.036918 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476e87faea1415d95ae014a93fa22af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.038292 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.038779 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49445f0f1dd1451c9ba066e61273f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.039671 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 7261088abbc641db8d737a48bbac25bd (DEFAULT_TABLE table=default.Table_49cdd819727f4ad381e0aae8fba8d201 [id=f7e1e9a608664cd6967ef0521d9cb671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.040179 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7261088abbc641db8d737a48bbac25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.041455 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.041976 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49445f0f1dd1451c9ba066e61273f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.044504 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (DEFAULT_TABLE table=default.Table_3f5415973b9641a3a612e9236aad1e92 [id=bd1950deb4274d86bf288d501cf5c710]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.045006 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.026726 28323 tablet_service.cc:1467] Processing CreateTablet for tablet b476e87faea1415d95ae014a93fa22af (DEFAULT_TABLE table=default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.048194 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476e87faea1415d95ae014a93fa22af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.041500 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 7261088abbc641db8d737a48bbac25bd (DEFAULT_TABLE table=default.Table_49cdd819727f4ad381e0aae8fba8d201 [id=f7e1e9a608664cd6967ef0521d9cb671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.056367 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7261088abbc641db8d737a48bbac25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.056660 28195 tablet_service.cc:1467] Processing CreateTablet for tablet b59c0bc115d949f69d80e1f3628d5a79 (DEFAULT_TABLE table=default.Table_63c1e1044be8466cb29bf56927fb925e [id=387303a065da4e2ebe6a28ebf492ae46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.057193 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59c0bc115d949f69d80e1f3628d5a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.060333 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 34238c9859724d97a32f2a1d9315edb1 (DEFAULT_TABLE table=default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.061085 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34238c9859724d97a32f2a1d9315edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.026917 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb (DEFAULT_TABLE table=default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a [id=f41fc50b1ca142f290aabb60f6cf0cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.076313 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de9e159d2ba47039dd6bd207a9cddeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.026793 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 34238c9859724d97a32f2a1d9315edb1 (DEFAULT_TABLE table=default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.080901 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34238c9859724d97a32f2a1d9315edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.083958 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 02d45788ae644a8581acca54cc55dd3f
I20240503 14:12:10.085317 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d45788ae644a8581acca54cc55dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.085402 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3d54cc5c03034a6685c1da34f9bd65ac
I20240503 14:12:10.085906 28027 catalog_manager.cc:6554] ReplaceTablet: tablet 3d54cc5c03034a6685c1da34f9bd65ac deleted and replaced by tablet d7c79ffee3954b5c870f4a8aa6379517
I20240503 14:12:10.086895 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.087627 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0bd5234791524eddb7edd493e39fb993
W20240503 14:12:10.087785 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3d54cc5c03034a6685c1da34f9bd65ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d54cc5c03034a6685c1da34f9bd65ac already in progress: creating tablet
I20240503 14:12:10.088188 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 0bd5234791524eddb7edd493e39fb993 deleted and replaced by tablet 9b21cd1262d94d1c994b29c44f86cb5e
I20240503 14:12:10.088512 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.089147 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd5234791524eddb7edd493e39fb993 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b21cd1262d94d1c994b29c44f86cb5e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.089459 28653 tablet_replica.cc:330] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:10.089749 28653 raft_consensus.cc:2229] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:10.089965 28653 raft_consensus.cc:2258] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.093092 28653 ts_tablet_manager.cc:1875] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:10.097533 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d45788ae644a8581acca54cc55dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.098959 29266 raft_consensus.cc:482] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.099148 29266 raft_consensus.cc:504] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.099896 29266 leader_election.cc:288] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.100392 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03bf91cf5414208b6767482223bc240" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:10.100714 28214 raft_consensus.cc:2379] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:10.106108 28653 ts_tablet_manager.cc:1888] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:10.108330 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd5234791524eddb7edd493e39fb993 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b21cd1262d94d1c994b29c44f86cb5e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.108860 28653 log.cc:1196] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0bd5234791524eddb7edd493e39fb993
I20240503 14:12:10.109432 28653 ts_tablet_manager.cc:1909] T 0bd5234791524eddb7edd493e39fb993 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:10.112403 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0bd5234791524eddb7edd493e39fb993 (table default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]) successfully deleted
W20240503 14:12:10.115489 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 02d45788ae644a8581acca54cc55dd3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d45788ae644a8581acca54cc55dd3f already in progress: creating tablet
I20240503 14:12:10.117147 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93462d2b1a474bd688d6266cd3b0695c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.118682 28489 raft_consensus.cc:2454] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:10.121773 29293 raft_consensus.cc:2726] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.010437 28614 tablet_bootstrap.cc:492] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:10.122551 28614 ts_tablet_manager.cc:1367] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.169s user 0.003s sys 0.002s
I20240503 14:12:10.123495 28614 raft_consensus.cc:348] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.123752 28614 raft_consensus.cc:374] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.123871 28614 raft_consensus.cc:729] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.124166 28614 consensus_queue.cc:260] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.128618 28029 catalog_manager.cc:5484] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:10.133240 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 6696a237c3dd428097e2ebdf24afd63f (DEFAULT_TABLE table=default.Table_43c9760cf3b54d68b218cafe8393cda2 [id=821a0387d6f44024b05959350fd53b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.133677 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.253:44259
--webserver_interface=127.27.50.253
--webserver_port=43185
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 14:12:10.134124 28337 tablet_service.cc:1467] Processing CreateTablet for tablet cf810e84367f420b928a171790849445 (DEFAULT_TABLE table=default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.134788 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a (DEFAULT_TABLE table=default.Table_4eec071da1eb4980b74b08ce5b21765b [id=50c1a152084e4b9690986b223b55e70a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.135130 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 33ee9b69dd1f47219e7d3126207810c7 (DEFAULT_TABLE table=default.Table_7a8f9e31244b4b7aaeb8ea4b458599cc [id=1bad537a243d4d3c86d1b2add999ebd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.135537 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 86f1ccc26f50432c9ef8789c01ef3519 (DEFAULT_TABLE table=default.Table_a41a77e2797147c680094d0b130290b8 [id=7bf58fd336ef42f6a2569b4c70332318]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.135843 28328 tablet_service.cc:1467] Processing CreateTablet for tablet a31555fc800142bd849dbd572a194002 (DEFAULT_TABLE table=default.Table_59366cab2245406f967358e760726e9b [id=5d99991da3ec4a0e9ea84e4bb6762014]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.135551 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet 75dafd98b05d4c63bbf30165a2ac8d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.136189 28187 tablet_service.cc:1467] Processing CreateTablet for tablet cf810e84367f420b928a171790849445 (DEFAULT_TABLE table=default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.136404 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 6696a237c3dd428097e2ebdf24afd63f (DEFAULT_TABLE table=default.Table_43c9760cf3b54d68b218cafe8393cda2 [id=821a0387d6f44024b05959350fd53b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.136677 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf810e84367f420b928a171790849445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.136933 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f (DEFAULT_TABLE table=default.Table_2a170feff0884500b519684ac43a06b0 [id=2d1a26ea051048589afec842f2a31c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.137364 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7779f7a9544e17aea7135eab2ccd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.137557 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 86f1ccc26f50432c9ef8789c01ef3519 (DEFAULT_TABLE table=default.Table_a41a77e2797147c680094d0b130290b8 [id=7bf58fd336ef42f6a2569b4c70332318]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.137917 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f1ccc26f50432c9ef8789c01ef3519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.139505 28193 tablet_service.cc:1467] Processing CreateTablet for tablet a31555fc800142bd849dbd572a194002 (DEFAULT_TABLE table=default.Table_59366cab2245406f967358e760726e9b [id=5d99991da3ec4a0e9ea84e4bb6762014]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.139971 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31555fc800142bd849dbd572a194002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.140410 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a (DEFAULT_TABLE table=default.Table_4eec071da1eb4980b74b08ce5b21765b [id=50c1a152084e4b9690986b223b55e70a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.140795 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfd6d40a57e4bea973874ff12c0ea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.144483 28191 tablet_service.cc:1467] Processing CreateTablet for tablet ac5189b6cdaa486c8c2f1b5c141d513b (DEFAULT_TABLE table=default.Table_922e97aa11034bbda5a51b53fc131880 [id=3c2f6c0cbdc645d3a76cc53a236687a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.144971 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5189b6cdaa486c8c2f1b5c141d513b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.151496 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03bf91cf5414208b6767482223bc240" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.153096 29287 raft_consensus.cc:482] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.155324 29287 raft_consensus.cc:504] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.157738 28025 catalog_manager.cc:5484] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:10.159346 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_f2493099c9fb43b7b2f070935a6a5f1f [id=084b186eba154ce1939f530379f4ce76]). Expected version 1 got 0
I20240503 14:12:10.159665 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:10.159870 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:10.160050 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 605531ed29214b1ca27011c5f5e96b11 (table default.Table_701e9c2bdea449d49f86dffb86090877 [id=651246fc28e144af9b8a0bbe9926aa22]). Expected version 1 got 0
W20240503 14:12:10.161532 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 605531ed29214b1ca27011c5f5e96b11 consensus state has no leader
I20240503 14:12:10.163924 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6b099d92a07649ee821a8507248767ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.164134 28150 leader_election.cc:302] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:10.164642 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b099d92a07649ee821a8507248767ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.164778 29243 raft_consensus.cc:2781] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.165132 29243 raft_consensus.cc:482] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.165251 29243 raft_consensus.cc:3037] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:10.165660 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 223038us
I20240503 14:12:10.166306 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6696a237c3dd428097e2ebdf24afd63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.168735 29287 leader_election.cc:288] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:10.169238 28180 tablet_service.cc:1467] Processing CreateTablet for tablet a035a5249f8348b591b27ba3fa7ae43a (DEFAULT_TABLE table=default.Table_848d74c659274863a2c1046f6de2568e [id=ed5de25a209e4d3f944021a08a498d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.169749 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a035a5249f8348b591b27ba3fa7ae43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.170321 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_11703c3db99a47cc85741ba32923f695" } modify_external_catalogs: true
I20240503 14:12:10.170552 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_11703c3db99a47cc85741ba32923f695" } modify_external_catalogs: true
I20240503 14:12:10.171979 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fe566740b43c42658f0a0ff2de59038a" } modify_external_catalogs: true
I20240503 14:12:10.172200 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fe566740b43c42658f0a0ff2de59038a" } modify_external_catalogs: true
I20240503 14:12:10.175024 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3949c7d2081d40b78d82c0b7a42e5dfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.175659 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3949c7d2081d40b78d82c0b7a42e5dfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.176798 28604 tablet_bootstrap.cc:492] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:10.181602 28182 tablet_service.cc:1467] Processing CreateTablet for tablet 646f83185d5241268d4126d10561e937 (DEFAULT_TABLE table=default.Table_4d256c29f9b7400497caee67285a9831 [id=6c9175a58b8544798ce769460dfd9cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.182045 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 7d348da459634561985c4ac1710c0bb2 (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.184319 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_55ff9f3cd6f84368add48d9cecc3fb1b" } new_table_name: "default.Table_87bcc8d1e5244ebdb6ed58c62b34f709" modify_external_catalogs: true
I20240503 14:12:10.185046 29281 raft_consensus.cc:482] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.169092 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 33ee9b69dd1f47219e7d3126207810c7 (DEFAULT_TABLE table=default.Table_7a8f9e31244b4b7aaeb8ea4b458599cc [id=1bad537a243d4d3c86d1b2add999ebd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.187130 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 (DEFAULT_TABLE table=default.Table_d7bd421e71724c8a89b7cd478f447a3b [id=e0a3f887fca8477ba19c4ed3738b1955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.187382 28604 ts_tablet_manager.cc:1367] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.510s user 0.043s sys 0.104s
I20240503 14:12:10.188057 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646f83185d5241268d4126d10561e937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.189050 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_64de235d5f5043b18820b85f48405374"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.189740 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64de235d5f5043b18820b85f48405374 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.190268 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.187511 29281 raft_consensus.cc:504] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.192893 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 1603e2360041427f8635782e812b6431 with delete_type TABLET_DATA_DELETED (replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.193509 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet da19787034c24ebe929601d49cb21b3b with delete_type TABLET_DATA_DELETED (replaced by tablet 7d348da459634561985c4ac1710c0bb2) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.193413 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cca3611a96024dd49da0111bbdf07abd" } new_table_name: "default.Table_9bcbc7e8d1154ffd94951feb50445bc2" modify_external_catalogs: true
I20240503 14:12:10.194154 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ddd791d4e14630bfb3b4b1513f76e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.194326 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d9985eecda4f3194c3251216aa58ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.194926 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.195356 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_848d74c659274863a2c1046f6de2568e" } modify_external_catalogs: true
I20240503 14:12:10.195626 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_79726aaee899484aaf8147ce414cbd87" } modify_external_catalogs: true
I20240503 14:12:10.195650 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_848d74c659274863a2c1046f6de2568e" } modify_external_catalogs: true
I20240503 14:12:10.195820 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_79726aaee899484aaf8147ce414cbd87" } modify_external_catalogs: true
I20240503 14:12:10.197467 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d348da459634561985c4ac1710c0bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.197887 28614 ts_tablet_manager.cc:1398] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:10.199173 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6696a237c3dd428097e2ebdf24afd63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.199211 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f (DEFAULT_TABLE table=default.Table_2a170feff0884500b519684ac43a06b0 [id=2d1a26ea051048589afec842f2a31c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.199739 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7779f7a9544e17aea7135eab2ccd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.202930 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ee9b69dd1f47219e7d3126207810c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.204370 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f1ccc26f50432c9ef8789c01ef3519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.206838 28320 tablet_service.cc:1467] Processing CreateTablet for tablet ac5189b6cdaa486c8c2f1b5c141d513b (DEFAULT_TABLE table=default.Table_922e97aa11034bbda5a51b53fc131880 [id=3c2f6c0cbdc645d3a76cc53a236687a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.207350 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5189b6cdaa486c8c2f1b5c141d513b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.208266 28334 tablet_service.cc:1467] Processing CreateTablet for tablet a035a5249f8348b591b27ba3fa7ae43a (DEFAULT_TABLE table=default.Table_848d74c659274863a2c1046f6de2568e [id=ed5de25a209e4d3f944021a08a498d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.208711 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfd6d40a57e4bea973874ff12c0ea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.208698 28290 leader_election.cc:302] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.208782 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7945118bb17b431dac8670007d2d0a18" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:10.209160 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 56312.5us
I20240503 14:12:10.210527 29266 raft_consensus.cc:3037] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.210824 28319 tablet_service.cc:1467] Processing CreateTablet for tablet 646f83185d5241268d4126d10561e937 (DEFAULT_TABLE table=default.Table_4d256c29f9b7400497caee67285a9831 [id=6c9175a58b8544798ce769460dfd9cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.211483 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 (DEFAULT_TABLE table=default.Table_d7bd421e71724c8a89b7cd478f447a3b [id=e0a3f887fca8477ba19c4ed3738b1955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.213251 29266 raft_consensus.cc:2726] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.213863 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74df7781cc6a4dd6b6fd78a728ffd71b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:10.214762 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a035a5249f8348b591b27ba3fa7ae43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.215242 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 7d348da459634561985c4ac1710c0bb2 (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.215763 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ddd791d4e14630bfb3b4b1513f76e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.216372 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf810e84367f420b928a171790849445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.216687 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646f83185d5241268d4126d10561e937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.214129 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
I20240503 14:12:10.227288 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d348da459634561985c4ac1710c0bb2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:10.227437 28150 leader_election.cc:341] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.192728 29281 leader_election.cc:288] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.231245 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31555fc800142bd849dbd572a194002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.236922 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ee9b69dd1f47219e7d3126207810c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.239075 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7945118bb17b431dac8670007d2d0a18" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.246510 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8e4c3260d34f48dba9b1270f6391cb17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.248759 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e4c3260d34f48dba9b1270f6391cb17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.253942 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8d1519e30ef54d12ab71e31903605531"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.254702 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d1519e30ef54d12ab71e31903605531 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.256788 29243 raft_consensus.cc:504] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.257508 29243 leader_election.cc:288] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.257665 29281 raft_consensus.cc:482] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.258616 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:10.260152 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b75036cdf9e04ea4aa767fd12016f177 consensus state has no leader
I20240503 14:12:10.260705 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.261364 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.265343 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8c6202333af943cbbc0d09553c1c193c
I20240503 14:12:10.266311 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_965ce30debbf4cc6a42b1fee93fb54de" } new_table_name: "default.Table_530c3b5891cd412d8b15b2c2b91de623" modify_external_catalogs: true
I20240503 14:12:10.266811 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a035a5249f8348b591b27ba3fa7ae43a
I20240503 14:12:10.267536 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc68335896e1489789a7ca2101fa68ec" } new_table_name: "default.Table_3a0134977dde48a0849c29c36487c742" modify_external_catalogs: true
I20240503 14:12:10.267745 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_961bb9bd6fec443589e047ead06c1526"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.268368 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_961bb9bd6fec443589e047ead06c1526 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.268342 29281 raft_consensus.cc:504] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.269116 29281 leader_election.cc:288] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.270673 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_618386316f9b46f9a7770c3f998769f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.271235 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_618386316f9b46f9a7770c3f998769f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.272864 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c7e5a76bf36545a5b53fc3d60512a521"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.273667 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00b52ea336524beea2f255c8720f7472 consensus state has no leader
I20240503 14:12:10.274134 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d1c711b946024f31b19e3a3f2db88a0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.274703 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1c711b946024f31b19e3a3f2db88a0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:10.278155 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a already in progress: creating tablet
W20240503 14:12:10.279868 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7e5a76bf36545a5b53fc3d60512a521 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.280328 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 93462d2b1a474bd688d6266cd3b0695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.280637 28653 tablet_replica.cc:330] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:10.280840 28653 raft_consensus.cc:2229] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:10.281040 28653 raft_consensus.cc:2258] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.281635 28653 ts_tablet_manager.cc:1875] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:10.284054 28600 tablet_bootstrap.cc:492] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.284317 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd5234791524eddb7edd493e39fb993 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b21cd1262d94d1c994b29c44f86cb5e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.284790 29281 raft_consensus.cc:482] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.284952 29281 raft_consensus.cc:504] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.285254 28600 ts_tablet_manager.cc:1367] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.279s user 0.026s sys 0.013s
I20240503 14:12:10.285728 29281 leader_election.cc:288] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.288901 28600 raft_consensus.cc:348] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:10.289369 28600 raft_consensus.cc:374] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.289466 28600 raft_consensus.cc:729] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.289742 28600 consensus_queue.cc:260] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:10.288693 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:10.290877 28653 ts_tablet_manager.cc:1888] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:10.291110 28653 log.cc:1196] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/93462d2b1a474bd688d6266cd3b0695c
I20240503 14:12:10.291685 28653 ts_tablet_manager.cc:1909] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:10.291800 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82" } modify_external_catalogs: true
I20240503 14:12:10.292055 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82" } modify_external_catalogs: true
W20240503 14:12:10.293123 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9325d5142af4b018354bf38370b61e0 consensus state has no leader
I20240503 14:12:10.293274 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 93462d2b1a474bd688d6266cd3b0695c (table default.Table_7712b0c57d054073ac3f31fa183e72f1 [id=48c3f3e039044d10a6cfa1b6bcd5ae79]) successfully deleted
W20240503 14:12:10.294001 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0bd5234791524eddb7edd493e39fb993 because the tablet was not found. No further retry: Not found: Tablet not found: 0bd5234791524eddb7edd493e39fb993
I20240503 14:12:10.295327 28196 tablet_service.cc:1467] Processing CreateTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.295811 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c79ffee3954b5c870f4a8aa6379517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.295799 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:10.301514 28600 ts_tablet_manager.cc:1398] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240503 14:12:10.302721 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.303473 28600 tablet_bootstrap.cc:492] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.304962 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6202333af943cbbc0d09553c1c193c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.305357 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:10.307945 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4084bd4fbf604042bb33ff03c3f8cc19 consensus state has no leader
I20240503 14:12:10.309654 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a7295c2d3bd34e20a69986aadc9c7017 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]). Expected version 1 got 0
W20240503 14:12:10.310639 28031 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet a7295c2d3bd34e20a69986aadc9c7017 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a7295c2d3bd34e20a69986aadc9c7017 consensus state has no leader
I20240503 14:12:10.317782 28600 tablet_bootstrap.cc:654] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.320247 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.322038 29266 raft_consensus.cc:482] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.324023 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fb79fcf30d485798c64c815e372dc1) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.324836 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0982b99a02ee48f2b6379e7be56c4c02
I20240503 14:12:10.325142 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 20b176a071ba4f3fbff2a64d6e46027a
I20240503 14:12:10.329134 28614 tablet_bootstrap.cc:492] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:10.330463 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b255d3e3dc64fb69c2ef6ed389db6a6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:10.330670 29297 raft_consensus.cc:482] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.328346 29266 raft_consensus.cc:504] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:10.331087 28150 leader_election.cc:341] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.333376 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a7295c2d3bd34e20a69986aadc9c7017 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]). Expected version 1 got 0
I20240503 14:12:10.329960 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93462d2b1a474bd688d6266cd3b0695c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:10.334379 28150 leader_election.cc:341] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.331295 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129717ef137c41c5a08d401c64164187" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:10.335549 28150 leader_election.cc:341] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.335862 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.336791 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e (DEFAULT_TABLE table=default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.341468 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b21cd1262d94d1c994b29c44f86cb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.337940 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.338110 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.336339 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 129717ef137c41c5a08d401c64164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.342988 28661 ts_tablet_manager.cc:1888] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:10.343480 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a035a5249f8348b591b27ba3fa7ae43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a035a5249f8348b591b27ba3fa7ae43a already in progress: creating tablet
W20240503 14:12:10.343662 28150 leader_election.cc:341] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.343861 28150 leader_election.cc:302] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.344255 29281 raft_consensus.cc:2726] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.346556 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.346988 29266 leader_election.cc:288] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:10.347576 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.352411 28653 tablet_replica.cc:330] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:10.352694 28653 raft_consensus.cc:2229] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:10.352869 28653 raft_consensus.cc:2258] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.354275 28600 tablet_bootstrap.cc:492] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.354414 28600 ts_tablet_manager.cc:1367] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.000s
I20240503 14:12:10.355105 28600 raft_consensus.cc:348] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.355353 28600 raft_consensus.cc:374] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.355407 28600 raft_consensus.cc:729] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.355806 29281 raft_consensus.cc:482] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.355933 29281 raft_consensus.cc:504] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:10.356511 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 8c6202333af943cbbc0d09553c1c193c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6202333af943cbbc0d09553c1c193c already in progress: creating tablet
I20240503 14:12:10.356575 28661 log.cc:1196] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/cae4969fc9474fdbaeae1b4d09e4875b
I20240503 14:12:10.356575 29281 leader_election.cc:288] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.357043 28661 ts_tablet_manager.cc:1909] T cae4969fc9474fdbaeae1b4d09e4875b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:10.355633 28600 consensus_queue.cc:260] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.358204 28661 tablet_replica.cc:330] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:10.358422 28661 raft_consensus.cc:2229] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:10.358747 28661 pending_rounds.cc:62] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:10.358815 28661 pending_rounds.cc:68] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597688682745856 op_type: NO_OP noop_request { }
I20240503 14:12:10.358963 28661 raft_consensus.cc:2866] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:10.359061 28661 raft_consensus.cc:2258] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.359251 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 007a892ef8504b848a400c05740eaa0c (DEFAULT_TABLE table=default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.359710 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007a892ef8504b848a400c05740eaa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.360020 28653 ts_tablet_manager.cc:1875] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:10.362303 29281 raft_consensus.cc:482] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.362485 29281 raft_consensus.cc:504] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.363121 29281 leader_election.cc:288] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.363191 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.363339 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb (DEFAULT_TABLE table=default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a [id=f41fc50b1ca142f290aabb60f6cf0cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.363792 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de9e159d2ba47039dd6bd207a9cddeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.364542 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605531ed29214b1ca27011c5f5e96b11" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:10.365089 28150 leader_election.cc:341] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.364902 28600 ts_tablet_manager.cc:1398] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:12:10.366299 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605531ed29214b1ca27011c5f5e96b11" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.366348 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129717ef137c41c5a08d401c64164187" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.366748 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b255d3e3dc64fb69c2ef6ed389db6a6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.367089 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd76060daab84c2f9f6727b8a9bb3a49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.367867 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 34238c9859724d97a32f2a1d9315edb1 (DEFAULT_TABLE table=default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.367751 28474 tablet_service.cc:1467] Processing CreateTablet for tablet d832cd30f2414303818d34a0cdf47e60 (DEFAULT_TABLE table=default.Table_a0b97d2f3edd4c549bbd9c1748f4d82f [id=566ce63bc6594fb29df2f8eb42c940e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.370288 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:10.370612 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:10.371358 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d832cd30f2414303818d34a0cdf47e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.372834 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f2493099c9fb43b7b2f070935a6a5f1f" } new_table_name: "default.Table_19f17d20edd24196b9efade6dca84369" modify_external_catalogs: true
I20240503 14:12:10.373227 28187 tablet_service.cc:1467] Processing CreateTablet for tablet b59945d011b64cba87c854e2ffde8226 (DEFAULT_TABLE table=default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.373639 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59945d011b64cba87c854e2ffde8226. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:10.374083 28150 leader_election.cc:341] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.374258 28150 leader_election.cc:302] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.374624 29243 consensus_queue.cc:1035] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.375154 29243 raft_consensus.cc:2726] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:10.375717 28150 leader_election.cc:341] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.375865 28150 leader_election.cc:302] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:10.376304 28150 leader_election.cc:341] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.376446 28150 leader_election.cc:302] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.376470 28460 tablet_service.cc:1467] Processing CreateTablet for tablet b59c0bc115d949f69d80e1f3628d5a79 (DEFAULT_TABLE table=default.Table_63c1e1044be8466cb29bf56927fb925e [id=387303a065da4e2ebe6a28ebf492ae46]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:10.376612 28150 raft_consensus.cc:2670] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:10.376833 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93462d2b1a474bd688d6266cd3b0695c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:10.376922 28150 leader_election.cc:341] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.377367 29281 raft_consensus.cc:2726] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.379578 28489 raft_consensus.cc:3037] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.379776 28600 tablet_bootstrap.cc:492] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.381345 28465 tablet_service.cc:1467] Processing CreateTablet for tablet b476e87faea1415d95ae014a93fa22af (DEFAULT_TABLE table=default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.383064 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 (DEFAULT_TABLE table=default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3 [id=ca041900898046468ff2eec0d0fadd85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.383610 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 0f223cbbcaa5402a888b084df0fe1c27 (DEFAULT_TABLE table=default.Table_6b099d92a07649ee821a8507248767ed [id=e6871aca8d244f00be81cf65dd32a18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.383741 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 7261088abbc641db8d737a48bbac25bd (DEFAULT_TABLE table=default.Table_49cdd819727f4ad381e0aae8fba8d201 [id=f7e1e9a608664cd6967ef0521d9cb671]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.384222 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 (DEFAULT_TABLE table=default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.384269 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86451bc448428da7485e70e0d54b89 (DEFAULT_TABLE table=default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.385550 28187 tablet_service.cc:1467] Processing CreateTablet for tablet e220336186f648419b186feada6f0124 (DEFAULT_TABLE table=default.Table_8e4c3260d34f48dba9b1270f6391cb17 [id=ffbda772b5ce4606914f5fecfff4a335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.386222 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34238c9859724d97a32f2a1d9315edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.386979 28600 tablet_bootstrap.cc:654] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.388291 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476e87faea1415d95ae014a93fa22af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.391486 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d45788ae644a8581acca54cc55dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.393906 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (DEFAULT_TABLE table=default.Table_3f5415973b9641a3a612e9236aad1e92 [id=bd1950deb4274d86bf288d501cf5c710]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.394490 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.394959 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49445f0f1dd1451c9ba066e61273f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:10.395733 28150 consensus_peers.cc:485] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:10.397047 28600 tablet_bootstrap.cc:492] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.397267 28600 ts_tablet_manager.cc:1367] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.001s
I20240503 14:12:10.398165 28600 raft_consensus.cc:348] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.398391 28600 raft_consensus.cc:374] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.398471 28600 raft_consensus.cc:729] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.398761 28600 consensus_queue.cc:260] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.399505 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7261088abbc641db8d737a48bbac25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.404217 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd5234791524eddb7edd493e39fb993 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b21cd1262d94d1c994b29c44f86cb5e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.404635 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.404716 28600 ts_tablet_manager.cc:1398] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:10.408130 28600 tablet_bootstrap.cc:492] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.409328 28661 ts_tablet_manager.cc:1875] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:10.409493 28493 raft_consensus.cc:1261] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Refusing update from remote peer a64bc15d65d84ad5a005960a9a84df9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:10.410398 29243 consensus_queue.cc:1035] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.411726 28600 tablet_bootstrap.cc:654] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.413511 28604 raft_consensus.cc:348] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.413862 28604 raft_consensus.cc:374] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.417380 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (table default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]). Expected version 1 got 0
I20240503 14:12:10.419797 28178 tablet_service.cc:1467] Processing CreateTablet for tablet ac752d8ab4fd4a218b052f388f416494 (DEFAULT_TABLE table=default.Table_c7e5a76bf36545a5b53fc3d60512a521 [id=e6889b2bf6214cabba6669c414c22f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.420111 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:10.420611 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:10.420951 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:10.421977 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 167a926fa61747e2b4fcb52ba4eeea5b (DEFAULT_TABLE table=default.Table_961bb9bd6fec443589e047ead06c1526 [id=8362eec57bb34b0e976ebed080aa4e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.423245 28457 tablet_service.cc:1467] Processing CreateTablet for tablet a31555fc800142bd849dbd572a194002 (DEFAULT_TABLE table=default.Table_59366cab2245406f967358e760726e9b [id=5d99991da3ec4a0e9ea84e4bb6762014]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.423146 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 6696a237c3dd428097e2ebdf24afd63f (DEFAULT_TABLE table=default.Table_43c9760cf3b54d68b218cafe8393cda2 [id=821a0387d6f44024b05959350fd53b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.424115 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 888636e9beca43edab96a8736751409e (DEFAULT_TABLE table=default.Table_d1c711b946024f31b19e3a3f2db88a0c [id=3854b08f7cd348c897aac4aacb7f5d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.426442 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd (DEFAULT_TABLE table=default.Table_8d1519e30ef54d12ab71e31903605531 [id=fa291ee5c8f54e89bcef6755f4ff3f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.427142 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (table default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]). Expected version 1 got 0
I20240503 14:12:10.427784 28188 tablet_service.cc:1467] Processing CreateTablet for tablet aac51b83abe14ab5b1fb5d627233e58e (DEFAULT_TABLE table=default.Table_618386316f9b46f9a7770c3f998769f5 [id=ffcb298fb844406881d9a8812e24e896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.431427 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9bcbc7e8d1154ffd94951feb50445bc2" } modify_external_catalogs: true
I20240503 14:12:10.431849 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9bcbc7e8d1154ffd94951feb50445bc2" } modify_external_catalogs: true
I20240503 14:12:10.433068 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.433357 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f223cbbcaa5402a888b084df0fe1c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.433873 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86451bc448428da7485e70e0d54b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.434371 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e220336186f648419b186feada6f0124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.434942 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac752d8ab4fd4a218b052f388f416494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.435385 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167a926fa61747e2b4fcb52ba4eeea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.435777 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888636e9beca43edab96a8736751409e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.436237 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdc868ddf0a490292eae78ebc8c8ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.440855 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac51b83abe14ab5b1fb5d627233e58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.441742 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_41a322f940d942adab752b8952a9f4a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.442440 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41a322f940d942adab752b8952a9f4a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.445551 28614 tablet_bootstrap.cc:654] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.453694 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6202333af943cbbc0d09553c1c193c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.454763 28604 raft_consensus.cc:729] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.346208 29297 raft_consensus.cc:504] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.459311 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 0f223cbbcaa5402a888b084df0fe1c27 (DEFAULT_TABLE table=default.Table_6b099d92a07649ee821a8507248767ed [id=e6871aca8d244f00be81cf65dd32a18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.460398 28327 tablet_service.cc:1467] Processing CreateTablet for tablet b59945d011b64cba87c854e2ffde8226 (DEFAULT_TABLE table=default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.461015 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:10.464246 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d1c711b946024f31b19e3a3f2db88a0c" } new_table_name: "default.Table_a2f9a3630d1d4bb79f9389f8e58257f5" modify_external_catalogs: true
I20240503 14:12:10.465793 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd (DEFAULT_TABLE table=default.Table_8d1519e30ef54d12ab71e31903605531 [id=fa291ee5c8f54e89bcef6755f4ff3f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.465860 28321 tablet_service.cc:1467] Processing CreateTablet for tablet aac51b83abe14ab5b1fb5d627233e58e (DEFAULT_TABLE table=default.Table_618386316f9b46f9a7770c3f998769f5 [id=ffcb298fb844406881d9a8812e24e896]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:10.465824 28290 leader_election.cc:341] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 40da5f5fff94462d8af740f3ee337e76
I20240503 14:12:10.466454 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 888636e9beca43edab96a8736751409e (DEFAULT_TABLE table=default.Table_d1c711b946024f31b19e3a3f2db88a0c [id=3854b08f7cd348c897aac4aacb7f5d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.466926 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd76060daab84c2f9f6727b8a9bb3a49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:10.467491 29243 raft_consensus.cc:482] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.467675 29243 raft_consensus.cc:504] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.468405 29243 leader_election.cc:288] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.472381 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e (DEFAULT_TABLE table=default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.474418 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_633e72d5fd384b689e09a7eef5a8d1d9" } modify_external_catalogs: true
I20240503 14:12:10.474668 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_633e72d5fd384b689e09a7eef5a8d1d9" } modify_external_catalogs: true
W20240503 14:12:10.476106 28150 consensus_peers.cc:485] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:10.476356 28150 leader_election.cc:341] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.476490 28150 leader_election.cc:302] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.476780 29297 leader_election.cc:288] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.476951 29266 raft_consensus.cc:482] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.477097 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ddcf99aec7384d15b0149124ff7a6646" } modify_external_catalogs: true
I20240503 14:12:10.477312 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ddcf99aec7384d15b0149124ff7a6646" } modify_external_catalogs: true
I20240503 14:12:10.478003 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86451bc448428da7485e70e0d54b89 (DEFAULT_TABLE table=default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.478776 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5bbe821329104389be37f7890f051041"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.479427 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bbe821329104389be37f7890f051041 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.480185 28336 tablet_service.cc:1467] Processing CreateTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.477398 29266 raft_consensus.cc:504] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:10.481683 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 888636e9beca43edab96a8736751409e consensus state has no leader
I20240503 14:12:10.482120 29243 raft_consensus.cc:2726] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.483389 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.483489 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e02fae7462a5462eb76e4e39a374b6e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.483655 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 167a926fa61747e2b4fcb52ba4eeea5b (DEFAULT_TABLE table=default.Table_961bb9bd6fec443589e047ead06c1526 [id=8362eec57bb34b0e976ebed080aa4e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.483906 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d5808426f2954b4981e680e8a56f5a5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.484377 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5808426f2954b4981e680e8a56f5a5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:10.484871 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e02fae7462a5462eb76e4e39a374b6e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.485241 28334 tablet_service.cc:1467] Processing CreateTablet for tablet e220336186f648419b186feada6f0124 (DEFAULT_TABLE table=default.Table_8e4c3260d34f48dba9b1270f6391cb17 [id=ffbda772b5ce4606914f5fecfff4a335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.486131 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f223cbbcaa5402a888b084df0fe1c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.487130 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_955617b294c94c299632de62b5973057" } new_table_name: "default.Table_07467425de964d5aa2a124eaba28268c" modify_external_catalogs: true
I20240503 14:12:10.487255 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59945d011b64cba87c854e2ffde8226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.487660 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f (DEFAULT_TABLE table=default.Table_2a170feff0884500b519684ac43a06b0 [id=2d1a26ea051048589afec842f2a31c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.489282 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40da5f5fff94462d8af740f3ee337e76" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.490679 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 (DEFAULT_TABLE table=default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3 [id=ca041900898046468ff2eec0d0fadd85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.486588 28325 tablet_service.cc:1467] Processing CreateTablet for tablet ac752d8ab4fd4a218b052f388f416494 (DEFAULT_TABLE table=default.Table_c7e5a76bf36545a5b53fc3d60512a521 [id=e6889b2bf6214cabba6669c414c22f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.491282 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514b79467dc42bc89fdacf8e39a000d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:10.491727 28290 leader_election.cc:341] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 75dafd98b05d4c63bbf30165a2ac8d1e
I20240503 14:12:10.493356 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac51b83abe14ab5b1fb5d627233e58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.493995 29266 leader_election.cc:288] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:10.494197 28150 leader_election.cc:341] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.494488 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.495013 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b21cd1262d94d1c994b29c44f86cb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.494930 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_07fdeadf61a045d8aeb377c2a3c4c1af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.495513 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07fdeadf61a045d8aeb377c2a3c4c1af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.497661 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_59366cab2245406f967358e760726e9b" } new_table_name: "default.Table_d7b3acfa64194593899187c496e64c66" modify_external_catalogs: true
I20240503 14:12:10.501636 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e9609112657f427cbeeb3ef8f28cddf0
I20240503 14:12:10.501850 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 00b52ea336524beea2f255c8720f7472
W20240503 14:12:10.503327 28290 leader_election.cc:341] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:10.503527 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 13c875ec8c5e49d1bb20a23dd7a21902
I20240503 14:12:10.503490 28290 leader_election.cc:302] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.504220 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet e9609112657f427cbeeb3ef8f28cddf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.504257 29266 raft_consensus.cc:2726] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.505879 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b52ea336524beea2f255c8720f7472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.506542 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c875ec8c5e49d1bb20a23dd7a21902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.507305 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c875ec8c5e49d1bb20a23dd7a21902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.508165 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b52ea336524beea2f255c8720f7472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.514761 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet e9609112657f427cbeeb3ef8f28cddf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.515215 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.518976 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e8411f474c247c2abfd0bd9682fb8a2 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:10.520679 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f64509bdd63e4324867d81972965e77b" } new_table_name: "default.Table_6c996bea4ba44fefbb3bc602c3c76f58" modify_external_catalogs: true
I20240503 14:12:10.523190 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_83e923042498440797a6b72edecd6277"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.523871 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83e923042498440797a6b72edecd6277 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.526248 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59c0bc115d949f69d80e1f3628d5a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.526585 28604 consensus_queue.cc:260] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.530917 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cdd0b8d98dd949578e4a4525c2666b1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.530979 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4f7b933efafa4a8080b087eff9410a90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.531539 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f7b933efafa4a8080b087eff9410a90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:10.532636 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdd0b8d98dd949578e4a4525c2666b1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.540174 28653 ts_tablet_manager.cc:1888] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:10.540367 28653 log.cc:1196] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0b255d3e3dc64fb69c2ef6ed389db6a6
I20240503 14:12:10.540876 28653 ts_tablet_manager.cc:1909] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:10.541869 28467 tablet_service.cc:1467] Processing CreateTablet for tablet cf810e84367f420b928a171790849445 (DEFAULT_TABLE table=default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.542804 28603 ts_tablet_manager.cc:1888] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:10.542994 28603 log.cc:1196] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/779f373ef12a4da091309fcf247b5693
I20240503 14:12:10.543373 28603 ts_tablet_manager.cc:1909] T 779f373ef12a4da091309fcf247b5693 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:10.543753 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdc868ddf0a490292eae78ebc8c8ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.544239 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167a926fa61747e2b4fcb52ba4eeea5b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:10.544719 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7218cefb13ea4983b2111ed5fd0c997b consensus state has no leader
I20240503 14:12:10.544716 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac752d8ab4fd4a218b052f388f416494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.544556 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_20718ac286c9405b871c43b98ac94f8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.545150 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86451bc448428da7485e70e0d54b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.545058 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4d256c29f9b7400497caee67285a9831" } modify_external_catalogs: true
W20240503 14:12:10.545178 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20718ac286c9405b871c43b98ac94f8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.545267 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4d256c29f9b7400497caee67285a9831" } modify_external_catalogs: true
I20240503 14:12:10.545619 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888636e9beca43edab96a8736751409e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.546013 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e220336186f648419b186feada6f0124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.546437 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c79ffee3954b5c870f4a8aa6379517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.578190 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514b79467dc42bc89fdacf8e39a000d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:10.578895 28150 leader_election.cc:341] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.579058 28150 leader_election.cc:302] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.579550 29243 raft_consensus.cc:2726] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:10.582437 28290 leader_election.cc:341] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 75dafd98b05d4c63bbf30165a2ac8d1e
I20240503 14:12:10.582603 28290 leader_election.cc:302] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.583047 29266 raft_consensus.cc:2726] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.583323 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6696a237c3dd428097e2ebdf24afd63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.583849 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 86f1ccc26f50432c9ef8789c01ef3519 (DEFAULT_TABLE table=default.Table_a41a77e2797147c680094d0b130290b8 [id=7bf58fd336ef42f6a2569b4c70332318]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.584108 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a (DEFAULT_TABLE table=default.Table_4eec071da1eb4980b74b08ce5b21765b [id=50c1a152084e4b9690986b223b55e70a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.584493 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfd6d40a57e4bea973874ff12c0ea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.584853 28459 tablet_service.cc:1467] Processing CreateTablet for tablet ac5189b6cdaa486c8c2f1b5c141d513b (DEFAULT_TABLE table=default.Table_922e97aa11034bbda5a51b53fc131880 [id=3c2f6c0cbdc645d3a76cc53a236687a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.585896 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 33ee9b69dd1f47219e7d3126207810c7 (DEFAULT_TABLE table=default.Table_7a8f9e31244b4b7aaeb8ea4b458599cc [id=1bad537a243d4d3c86d1b2add999ebd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.586997 28463 tablet_service.cc:1467] Processing CreateTablet for tablet a035a5249f8348b591b27ba3fa7ae43a (DEFAULT_TABLE table=default.Table_848d74c659274863a2c1046f6de2568e [id=ed5de25a209e4d3f944021a08a498d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.588011 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 7d348da459634561985c4ac1710c0bb2 (DEFAULT_TABLE table=default.Table_f64509bdd63e4324867d81972965e77b [id=7552d7d498de405fbddad0211f40e0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.589404 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 93462d2b1a474bd688d6266cd3b0695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.589896 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 646f83185d5241268d4126d10561e937 (DEFAULT_TABLE table=default.Table_4d256c29f9b7400497caee67285a9831 [id=6c9175a58b8544798ce769460dfd9cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.584904 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31555fc800142bd849dbd572a194002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.586985 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1aa939aced4556ae0f0de0e62d1f97" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.596865 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5189b6cdaa486c8c2f1b5c141d513b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.597194 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a035a5249f8348b591b27ba3fa7ae43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.597412 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d348da459634561985c4ac1710c0bb2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:10.602295 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a31555fc800142bd849dbd572a194002 consensus state has no leader
I20240503 14:12:10.605383 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 4e8411f474c247c2abfd0bd9682fb8a2
I20240503 14:12:10.605999 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 4e8411f474c247c2abfd0bd9682fb8a2 deleted and replaced by tablet a1ebec4622ef4950bd1c70451958c59b
I20240503 14:12:10.606261 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_285ab08f9ab94ca6a053a091945582ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.606879 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_285ab08f9ab94ca6a053a091945582ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.607946 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.608214 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e4c32619dd494186a5a26bbc9cc04dbe" } modify_external_catalogs: true
I20240503 14:12:10.608422 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e4c32619dd494186a5a26bbc9cc04dbe" } modify_external_catalogs: true
I20240503 14:12:10.608932 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_13c7175b4f704263a322c205dfb42854" } new_table_name: "default.Table_f6fdd56971514ba4a525832dbcd2bbf7" modify_external_catalogs: true
W20240503 14:12:10.610663 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d348da459634561985c4ac1710c0bb2 consensus state has no leader
I20240503 14:12:10.616333 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7779f7a9544e17aea7135eab2ccd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.616609 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 646f83185d5241268d4126d10561e937
I20240503 14:12:10.618484 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 646f83185d5241268d4126d10561e937 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.618844 28604 ts_tablet_manager.cc:1398] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.394s user 0.002s sys 0.002s
I20240503 14:12:10.619856 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f1ccc26f50432c9ef8789c01ef3519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.621196 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646f83185d5241268d4126d10561e937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.622851 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ee9b69dd1f47219e7d3126207810c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.625388 28604 tablet_bootstrap.cc:492] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:10.628042 29243 raft_consensus.cc:482] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.628180 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf810e84367f420b928a171790849445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.628342 29243 raft_consensus.cc:504] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.629050 29243 leader_election.cc:288] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.629968 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265dc5f2630e453ea71abddb1a9753f3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:10.630669 28150 leader_election.cc:341] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.640246 28653 tablet_replica.cc:330] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:10.640527 28653 raft_consensus.cc:2229] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:10.640750 28653 raft_consensus.cc:2258] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.641362 28653 ts_tablet_manager.cc:1875] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:10.645246 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d43823da09bb49be8f5d1a1149a4b11d consensus state has no leader
I20240503 14:12:10.646548 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 (table default.Table_068cea3b090241c7980d0edb57c05aec [id=80b934d1638444ebb855ebacf917c23e]) successfully deleted
I20240503 14:12:10.655889 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a186e8c3c3b440bba034806014b91bb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.656643 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a186e8c3c3b440bba034806014b91bb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.672269 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265dc5f2630e453ea71abddb1a9753f3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:10.673235 28150 leader_election.cc:341] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.673444 28150 leader_election.cc:302] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.673966 29243 raft_consensus.cc:2726] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.674338 28290 leader_election.cc:302] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.675015 29266 raft_consensus.cc:2726] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:12:10.675522 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 76373.4us
I20240503 14:12:10.676244 28603 tablet_replica.cc:330] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:10.676489 28603 raft_consensus.cc:2229] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:10.676709 28603 raft_consensus.cc:2258] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.677320 28603 ts_tablet_manager.cc:1875] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:10.677935 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet 1603e2360041427f8635782e812b6431 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:10.680106 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 779f373ef12a4da091309fcf247b5693 (table default.Table_8c90a0cf9e1f495ea0ea85f1f8726f52 [id=857b2f49969d4291b98de4f3776e627d]) successfully deleted
W20240503 14:12:10.681182 28290 consensus_peers.cc:485] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da19787034c24ebe929601d49cb21b3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:10.681643 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 646f83185d5241268d4126d10561e937 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.685395 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7ad492405f92420ba1dd8dcf793d915c" } new_table_name: "default.Table_354b5d04e8994484bf17eda27e660238" modify_external_catalogs: true
I20240503 14:12:10.685864 29266 raft_consensus.cc:482] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.686033 29266 raft_consensus.cc:504] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.686738 29266 leader_election.cc:288] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:10.687551 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.691722 28178 tablet_service.cc:1467] Processing CreateTablet for tablet d6f5103b31ed45b480252dcf94b72f9c (DEFAULT_TABLE table=default.Table_41a322f940d942adab752b8952a9f4a4 [id=18b53c1cdfb4468296f5bad79e15fb01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.694178 28489 raft_consensus.cc:2454] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:10.694845 28150 leader_election.cc:302] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:10.695134 28150 raft_consensus.cc:2670] T 93462d2b1a474bd688d6266cd3b0695c P a64bc15d65d84ad5a005960a9a84df9c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:10.697337 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 49445f0f1dd1451c9ba066e61273f2f1
I20240503 14:12:10.699214 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76]). Expected version 2 got 0
W20240503 14:12:10.701512 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e680f189add4b5c96de4046321aad2d consensus state has no leader
I20240503 14:12:10.703604 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.705122 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_43c9760cf3b54d68b218cafe8393cda2" } modify_external_catalogs: true
I20240503 14:12:10.705369 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_43c9760cf3b54d68b218cafe8393cda2" } modify_external_catalogs: true
I20240503 14:12:10.707584 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]). Expected version 1 got 0
I20240503 14:12:10.709571 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_41a322f940d942adab752b8952a9f4a4" } modify_external_catalogs: true
I20240503 14:12:10.709812 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_41a322f940d942adab752b8952a9f4a4" } modify_external_catalogs: true
I20240503 14:12:10.712217 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8d1519e30ef54d12ab71e31903605531" } modify_external_catalogs: true
I20240503 14:12:10.712464 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8d1519e30ef54d12ab71e31903605531" } modify_external_catalogs: true
W20240503 14:12:10.712752 28290 leader_election.cc:341] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6dab0296a0894a63afc16ed466c591d5
I20240503 14:12:10.713141 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.715232 28334 tablet_service.cc:1467] Processing CreateTablet for tablet d6f5103b31ed45b480252dcf94b72f9c (DEFAULT_TABLE table=default.Table_41a322f940d942adab752b8952a9f4a4 [id=18b53c1cdfb4468296f5bad79e15fb01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.720218 28604 tablet_bootstrap.cc:654] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.726779 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f5103b31ed45b480252dcf94b72f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.727607 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_87bf7222c4444f8f90511d56ac011976"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.728319 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87bf7222c4444f8f90511d56ac011976 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.728971 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dab0296a0894a63afc16ed466c591d5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.733369 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d6f5103b31ed45b480252dcf94b72f9c
I20240503 14:12:10.734076 28614 tablet_bootstrap.cc:492] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:10.734253 28614 ts_tablet_manager.cc:1367] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.414s user 0.018s sys 0.009s
I20240503 14:12:10.735069 28614 raft_consensus.cc:348] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.735311 28614 raft_consensus.cc:374] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.735384 28614 raft_consensus.cc:729] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.735749 28614 consensus_queue.cc:260] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.736678 28614 ts_tablet_manager.cc:1398] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:10.741685 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6696a237c3dd428097e2ebdf24afd63f
I20240503 14:12:10.743121 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd
I20240503 14:12:10.743297 28614 tablet_bootstrap.cc:492] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:10.744912 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f5103b31ed45b480252dcf94b72f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.745931 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f5103b31ed45b480252dcf94b72f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.748282 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.749272 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:10.754359 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 (DEFAULT_TABLE table=default.Table_d7bd421e71724c8a89b7cd478f447a3b [id=e0a3f887fca8477ba19c4ed3738b1955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.754930 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ddd791d4e14630bfb3b4b1513f76e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.757161 28614 tablet_bootstrap.cc:654] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.762194 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.765267 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cb1bb0240e554fb2a420db2ec7d00e68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.766008 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb1bb0240e554fb2a420db2ec7d00e68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.767465 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_69e2f4400fe2428ea429f0d666b9610c" } new_table_name: "default.Table_85dbdd19a2514c63bc5ee54794385f56" modify_external_catalogs: true
W20240503 14:12:10.778609 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 240abf8c2b2343408fc98a8dac1fdf09 consensus state has no leader
I20240503 14:12:10.784858 29113 raft_consensus.cc:482] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.785074 29113 raft_consensus.cc:504] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.785789 29113 leader_election.cc:288] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:10.788477 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6371e68ee85a47cdb0e8e21f7124d94a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:10.789278 28150 leader_election.cc:341] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.778046 29305 tablet.cc:1722] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:10.793566 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.794436 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.795723 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f5103b31ed45b480252dcf94b72f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:10.766542 28489 raft_consensus.cc:2454] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:10.785132 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6202333af943cbbc0d09553c1c193c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.801735 28290 leader_election.cc:302] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:10.787849 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.789463 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6371e68ee85a47cdb0e8e21f7124d94a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:10.803164 29266 raft_consensus.cc:2781] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.803295 29266 raft_consensus.cc:482] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:12:10.803231 28150 leader_election.cc:341] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:10.803392 29266 raft_consensus.cc:3037] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.803426 28150 leader_election.cc:302] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:10.805101 28600 tablet_bootstrap.cc:492] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.805306 28600 ts_tablet_manager.cc:1367] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.398s user 0.050s sys 0.036s
I20240503 14:12:10.806267 28600 raft_consensus.cc:348] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.808916 28600 raft_consensus.cc:374] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.809104 28600 raft_consensus.cc:729] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.808854 29281 raft_consensus.cc:2726] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:10.809149 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4fc8bf5e30814b5eb9e5ea28422b1802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.809484 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b76d17fed23441d3acf329eaa3d64853" } new_table_name: "default.Table_ea00e82e486041f49201fc0ef12fdd2d" modify_external_catalogs: true
I20240503 14:12:10.809454 28600 consensus_queue.cc:260] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.810068 28600 ts_tablet_manager.cc:1398] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:12:10.810081 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_34588b8057d642378dc78cac201ee75f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.810824 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0f214d103abd4357829af22666b5f9a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.811142 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6dfb1a277c824cbfad52ce51179289a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.812789 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f2b1cc65e0f5452fbc24389d16be4eb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.814843 28600 tablet_bootstrap.cc:492] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:10.814831 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f214d103abd4357829af22666b5f9a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:10.814981 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dfb1a277c824cbfad52ce51179289a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:10.816062 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34588b8057d642378dc78cac201ee75f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:10.816223 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2b1cc65e0f5452fbc24389d16be4eb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.816844 28600 tablet_bootstrap.cc:654] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:10.819880 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fc8bf5e30814b5eb9e5ea28422b1802 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.820497 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c355201f2d654091bbf81fd66561a942" } new_table_name: "default.Table_66a25fdf1a2147e8ba24818e5bdbabc4" modify_external_catalogs: true
I20240503 14:12:10.821699 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76]). Expected version 2 got 0
I20240503 14:12:10.823910 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e3372d35024401296239c262dc5fc5b" } modify_external_catalogs: true
I20240503 14:12:10.824204 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e3372d35024401296239c262dc5fc5b" } modify_external_catalogs: true
I20240503 14:12:10.832027 28600 tablet_bootstrap.cc:492] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.832293 28600 ts_tablet_manager.cc:1367] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240503 14:12:10.832337 28614 tablet_bootstrap.cc:492] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:10.832590 28614 ts_tablet_manager.cc:1367] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.004s
I20240503 14:12:10.833225 28600 raft_consensus.cc:348] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.833467 28600 raft_consensus.cc:374] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.833581 28600 raft_consensus.cc:729] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.833743 28614 raft_consensus.cc:348] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.834014 28614 raft_consensus.cc:374] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.833918 28600 consensus_queue.cc:260] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.834122 28614 raft_consensus.cc:729] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.834359 28614 consensus_queue.cc:260] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.835137 28614 ts_tablet_manager.cc:1398] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.836287 29266 raft_consensus.cc:504] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.840057 29266 leader_election.cc:288] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:10.836023 28614 tablet_bootstrap.cc:492] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:10.841087 28600 ts_tablet_manager.cc:1398] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:12:10.841413 28600 tablet_bootstrap.cc:492] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.842892 28614 tablet_bootstrap.cc:654] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.843091 28600 tablet_bootstrap.cc:654] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:10.844074 28290 leader_election.cc:341] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6dab0296a0894a63afc16ed466c591d5
I20240503 14:12:10.849857 28653 ts_tablet_manager.cc:1888] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:10.850019 28653 log.cc:1196] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/bd76060daab84c2f9f6727b8a9bb3a49
I20240503 14:12:10.850478 28653 ts_tablet_manager.cc:1909] T bd76060daab84c2f9f6727b8a9bb3a49 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:10.859730 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c924a74ee836406fb3157baef6f4c901 consensus state has no leader
I20240503 14:12:10.868060 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.869947 28600 tablet_bootstrap.cc:492] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.870172 28600 ts_tablet_manager.cc:1367] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.004s
I20240503 14:12:10.871054 28600 raft_consensus.cc:348] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.871318 28600 raft_consensus.cc:374] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.871402 28600 raft_consensus.cc:729] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.871667 28600 consensus_queue.cc:260] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.873790 28600 ts_tablet_manager.cc:1398] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:10.874188 28600 tablet_bootstrap.cc:492] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.873932 29287 consensus_queue.cc:1035] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240503 14:12:10.875942 28600 tablet_bootstrap.cc:654] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.859998 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3ede1f6683664f5d9956c6f825d3d7c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:10.876425 28455 tablet_service.cc:1467] Processing CreateTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.881366 28614 tablet_bootstrap.cc:492] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:10.881579 28614 ts_tablet_manager.cc:1367] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20240503 14:12:10.882483 28614 raft_consensus.cc:348] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.882748 28614 raft_consensus.cc:374] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.882835 28614 raft_consensus.cc:729] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.883152 28614 consensus_queue.cc:260] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.883689 28614 ts_tablet_manager.cc:1398] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:10.884065 28614 tablet_bootstrap.cc:492] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:10.885198 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e (DEFAULT_TABLE table=default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.888522 28614 tablet_bootstrap.cc:654] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.891275 28468 tablet_service.cc:1467] Processing CreateTablet for tablet b59945d011b64cba87c854e2ffde8226 (DEFAULT_TABLE table=default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.893293 28600 tablet_bootstrap.cc:492] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.893296 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dab0296a0894a63afc16ed466c591d5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:10.893510 28600 ts_tablet_manager.cc:1367] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 14:12:10.893579 28489 raft_consensus.cc:3037] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.894474 28600 raft_consensus.cc:348] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.894733 28600 raft_consensus.cc:374] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.894865 28600 raft_consensus.cc:729] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.895232 28600 consensus_queue.cc:260] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:10.895272 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ede1f6683664f5d9956c6f825d3d7c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.896107 28600 ts_tablet_manager.cc:1398] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:10.896454 28600 tablet_bootstrap.cc:492] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.898608 28600 tablet_bootstrap.cc:654] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.902957 28466 tablet_service.cc:1467] Processing CreateTablet for tablet e220336186f648419b186feada6f0124 (DEFAULT_TABLE table=default.Table_8e4c3260d34f48dba9b1270f6391cb17 [id=ffbda772b5ce4606914f5fecfff4a335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.904831 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:10.916170 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 8b1728b6f0e64661aaa8184ce51366d2 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:10.917471 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_48701243e80449bf86e7e0c400cebdc4" } new_table_name: "default.Table_193453f3dc0a440db9aaeada83e07e1a" modify_external_catalogs: true
I20240503 14:12:10.918087 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_446afb3f188b43f1aea50e0337a2c793"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.918788 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_446afb3f188b43f1aea50e0337a2c793 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.920630 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:10.921281 28600 tablet_bootstrap.cc:492] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:10.921926 28600 ts_tablet_manager.cc:1367] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
W20240503 14:12:10.922436 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f0795253db548fb84c2d7bb3dc48335 consensus state has no leader
I20240503 14:12:10.922401 28334 tablet_service.cc:1467] Processing CreateTablet for tablet c00dbc5a90674873b61b377e3e8d92fb (DEFAULT_TABLE table=default.Table_a186e8c3c3b440bba034806014b91bb6 [id=822d52647da14f238b5c547d4bbd9dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.922899 28600 raft_consensus.cc:348] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.923173 28600 raft_consensus.cc:374] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.923132 28331 tablet_service.cc:1467] Processing CreateTablet for tablet f8fe462c9ee54a27b9714f6457036d76 (DEFAULT_TABLE table=default.Table_285ab08f9ab94ca6a053a091945582ae [id=bd634bbb45704481a0476a7dfc8f0db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.923277 28600 raft_consensus.cc:729] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.923609 28600 consensus_queue.cc:260] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:10.924741 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5264956c580441a6a23c96d39a41ea71" } modify_external_catalogs: true
I20240503 14:12:10.924975 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5264956c580441a6a23c96d39a41ea71" } modify_external_catalogs: true
I20240503 14:12:10.925993 28600 ts_tablet_manager.cc:1398] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:10.926483 28600 tablet_bootstrap.cc:492] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:10.926390 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7f90c0c383146e393eb5890a2ce9168" } new_table_name: "default.Table_1145b045780c4d1e8fedd333eea661d4" modify_external_catalogs: true
I20240503 14:12:10.929662 28180 tablet_service.cc:1467] Processing CreateTablet for tablet f8fe462c9ee54a27b9714f6457036d76 (DEFAULT_TABLE table=default.Table_285ab08f9ab94ca6a053a091945582ae [id=bd634bbb45704481a0476a7dfc8f0db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.929929 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7b805aee554f8fa4c198ce3409c6e6 (DEFAULT_TABLE table=default.Table_83e923042498440797a6b72edecd6277 [id=4df57b62a5b34ac182e799495a848041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.930370 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7b805aee554f8fa4c198ce3409c6e6 (DEFAULT_TABLE table=default.Table_83e923042498440797a6b72edecd6277 [id=4df57b62a5b34ac182e799495a848041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.930958 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7b6df20b14d1ead036cdc42174d3a (DEFAULT_TABLE table=default.Table_cdd0b8d98dd949578e4a4525c2666b1c [id=0edef09212734f47be4ac77cca4fc186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.931429 28188 tablet_service.cc:1467] Processing CreateTablet for tablet ea261d8d5d174bee804aa2e171f35008 (DEFAULT_TABLE table=default.Table_4f7b933efafa4a8080b087eff9410a90 [id=cf499dac577641bd8cd74d615076204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.931494 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cca08ca8fa2f4f87a3793867e49f7b6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:10.932209 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cca08ca8fa2f4f87a3793867e49f7b6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:10.932498 28600 tablet_bootstrap.cc:654] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.935788 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 2b52ef7ecedf4273939406581d843980 (DEFAULT_TABLE table=default.Table_07fdeadf61a045d8aeb377c2a3c4c1af [id=be2dbf4734c74e93ad430d203979e633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.938334 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 3f81bbdb2ee149c5a446c608034585fe (DEFAULT_TABLE table=default.Table_e02fae7462a5462eb76e4e39a374b6e1 [id=0a3c96abf47b4f7cb543422990ae8812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.939136 28335 tablet_service.cc:1467] Processing CreateTablet for tablet e45df464f6954a56bec8b107b350320c (DEFAULT_TABLE table=default.Table_d5808426f2954b4981e680e8a56f5a5b [id=beabb84faa9f4d34a0293ea09c93e19a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.941293 28192 tablet_service.cc:1467] Processing CreateTablet for tablet e45df464f6954a56bec8b107b350320c (DEFAULT_TABLE table=default.Table_d5808426f2954b4981e680e8a56f5a5b [id=beabb84faa9f4d34a0293ea09c93e19a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.945282 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea261d8d5d174bee804aa2e171f35008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.946164 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac7b6df20b14d1ead036cdc42174d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.946691 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7b805aee554f8fa4c198ce3409c6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.947213 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b52ef7ecedf4273939406581d843980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.947757 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f81bbdb2ee149c5a446c608034585fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.948320 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45df464f6954a56bec8b107b350320c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.950701 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fe462c9ee54a27b9714f6457036d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.953529 28614 tablet_bootstrap.cc:492] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:10.953814 28614 ts_tablet_manager.cc:1367] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.003s
I20240503 14:12:10.954886 28614 raft_consensus.cc:348] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.955147 28614 raft_consensus.cc:374] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.955276 28614 raft_consensus.cc:729] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.955595 28614 consensus_queue.cc:260] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:10.959848 28614 ts_tablet_manager.cc:1398] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:10.968050 28614 tablet_bootstrap.cc:492] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:10.970043 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76]). Expected version 2 got 0
I20240503 14:12:10.972457 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0" } modify_external_catalogs: true
I20240503 14:12:10.973901 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0" } modify_external_catalogs: true
I20240503 14:12:10.976717 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 3f81bbdb2ee149c5a446c608034585fe (DEFAULT_TABLE table=default.Table_e02fae7462a5462eb76e4e39a374b6e1 [id=0a3c96abf47b4f7cb543422990ae8812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.977869 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 08cc84cf703048148abfe03b7dee12ce (DEFAULT_TABLE table=default.Table_20718ac286c9405b871c43b98ac94f8d [id=51930b6b93bd406581a479188de3423d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.978008 28614 tablet_bootstrap.cc:654] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.978644 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 2b52ef7ecedf4273939406581d843980 (DEFAULT_TABLE table=default.Table_07fdeadf61a045d8aeb377c2a3c4c1af [id=be2dbf4734c74e93ad430d203979e633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.980453 28327 tablet_service.cc:1467] Processing CreateTablet for tablet ea261d8d5d174bee804aa2e171f35008 (DEFAULT_TABLE table=default.Table_4f7b933efafa4a8080b087eff9410a90 [id=cf499dac577641bd8cd74d615076204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.982872 28336 tablet_service.cc:1467] Processing CreateTablet for tablet a1ebec4622ef4950bd1c70451958c59b (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.986167 28325 tablet_service.cc:1467] Processing CreateTablet for tablet a26e8389639748ae99832d7a7a9d5af3 (DEFAULT_TABLE table=default.Table_5bbe821329104389be37f7890f051041 [id=78e7487e239c4115ae6f3a3a45ca8b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.988477 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 08cc84cf703048148abfe03b7dee12ce (DEFAULT_TABLE table=default.Table_20718ac286c9405b871c43b98ac94f8d [id=51930b6b93bd406581a479188de3423d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.990739 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7b6df20b14d1ead036cdc42174d3a (DEFAULT_TABLE table=default.Table_cdd0b8d98dd949578e4a4525c2666b1c [id=0edef09212734f47be4ac77cca4fc186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.991020 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cc84cf703048148abfe03b7dee12ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.992801 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00dbc5a90674873b61b377e3e8d92fb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:10.993113 28474 consensus_meta.cc:287] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: Time spent flushing consensus metadata: real 0.513s user 0.004s sys 0.001s
I20240503 14:12:10.993160 28182 tablet_service.cc:1467] Processing CreateTablet for tablet a1ebec4622ef4950bd1c70451958c59b (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.993472 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fe462c9ee54a27b9714f6457036d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.993636 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebec4622ef4950bd1c70451958c59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.994086 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7b805aee554f8fa4c198ce3409c6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.994810 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45df464f6954a56bec8b107b350320c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.996277 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebec4622ef4950bd1c70451958c59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.996893 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26e8389639748ae99832d7a7a9d5af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.997527 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cc84cf703048148abfe03b7dee12ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.998167 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f81bbdb2ee149c5a446c608034585fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.999006 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 0f223cbbcaa5402a888b084df0fe1c27 (DEFAULT_TABLE table=default.Table_6b099d92a07649ee821a8507248767ed [id=e6871aca8d244f00be81cf65dd32a18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.999468 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b52ef7ecedf4273939406581d843980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.000110 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea261d8d5d174bee804aa2e171f35008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.006022 28614 tablet_bootstrap.cc:492] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:11.006527 28614 ts_tablet_manager.cc:1367] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.006s
W20240503 14:12:11.009462 28019 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 77f0081cc2cb48aaa368741109032fd1 (table default.Table_730611e6ad48480eb3f4498f3e79725b [id=ddb1f8d9ceae4da98dcb259cde8eac8c], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 77f0081cc2cb48aaa368741109032fd1 consensus state has no leader
I20240503 14:12:11.008615 28614 raft_consensus.cc:348] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.010435 28614 raft_consensus.cc:374] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.010882 28614 raft_consensus.cc:729] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.012125 28614 consensus_queue.cc:260] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.017556 28614 ts_tablet_manager.cc:1398] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:12:11.045343 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 (DEFAULT_TABLE table=default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3 [id=ca041900898046468ff2eec0d0fadd85]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.046924 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 65026.9us
I20240503 14:12:11.053189 29266 raft_consensus.cc:482] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.053651 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7c4e797497e74bbe98ff720b9ffd05ff" } modify_external_catalogs: true
I20240503 14:12:11.053910 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7c4e797497e74bbe98ff720b9ffd05ff" } modify_external_catalogs: true
W20240503 14:12:11.055490 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0926e523ef454a2089f4d40a328c1c6d consensus state has no leader
I20240503 14:12:11.057598 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280" } modify_external_catalogs: true
I20240503 14:12:11.057878 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280" } modify_external_catalogs: true
I20240503 14:12:11.059499 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a7295c2d3bd34e20a69986aadc9c7017
I20240503 14:12:11.062559 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1d2243e88c784b929de2fc4d94804219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:11.063334 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d2243e88c784b929de2fc4d94804219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.065740 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8b1728b6f0e64661aaa8184ce51366d2
I20240503 14:12:11.067968 28025 catalog_manager.cc:6554] ReplaceTablet: tablet 8b1728b6f0e64661aaa8184ce51366d2 deleted and replaced by tablet d4c8a97076a44f9eaa3a15026f90a304
I20240503 14:12:11.068385 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c5e3649c71f4d4da710c69f85a82c25 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:11.073175 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4f7b933efafa4a8080b087eff9410a90" } new_table_name: "default.Table_334a9c3578d445778df17229f9d401f7" modify_external_catalogs: true
I20240503 14:12:11.075117 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_30902594a17047d0ac5f2784f7a1d2eb" } new_table_name: "default.Table_2e88623765c84939b082bcf13612fd61" modify_external_catalogs: true
W20240503 14:12:11.077463 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a081ba59ad44890bf2f94874c034277 consensus state has no leader
I20240503 14:12:11.088127 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_021c165af8934008b2bb8bb0487c0af4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:11.089093 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_021c165af8934008b2bb8bb0487c0af4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.096625 29316 raft_consensus.cc:482] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:11.100028 28462 consensus_meta.cc:287] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc: Time spent flushing consensus metadata: real 0.576s user 0.001s sys 0.004s
I20240503 14:12:11.104802 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0a50b3e199314bfc8465fdb7b2425d07
W20240503 14:12:11.113523 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 67172us
I20240503 14:12:11.114871 28178 tablet_service.cc:1467] Processing CreateTablet for tablet c00dbc5a90674873b61b377e3e8d92fb (DEFAULT_TABLE table=default.Table_a186e8c3c3b440bba034806014b91bb6 [id=822d52647da14f238b5c547d4bbd9dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.105289 29316 raft_consensus.cc:504] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.116937 29243 raft_consensus.cc:482] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.117738 29281 raft_consensus.cc:482] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.117924 29281 raft_consensus.cc:504] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:11.119148 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b5f59b1cb2f440d4ae381befe0809756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:11.119740 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7295c2d3bd34e20a69986aadc9c7017" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:11.121184 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5f59b1cb2f440d4ae381befe0809756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:11.124958 28150 leader_election.cc:341] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.125499 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00dbc5a90674873b61b377e3e8d92fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.125555 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f7bbe9e54e404d698feb87bac95370d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:11.127593 28195 tablet_service.cc:1467] Processing CreateTablet for tablet a26e8389639748ae99832d7a7a9d5af3 (DEFAULT_TABLE table=default.Table_5bbe821329104389be37f7890f051041 [id=78e7487e239c4115ae6f3a3a45ca8b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.128355 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26e8389639748ae99832d7a7a9d5af3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:11.128350 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7bbe9e54e404d698feb87bac95370d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.124591 29243 raft_consensus.cc:504] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.164503 29281 leader_election.cc:288] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:11.171471 29243 leader_election.cc:288] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:11.179368 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b21cd1262d94d1c994b29c44f86cb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.180626 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 7b86451bc448428da7485e70e0d54b89 (DEFAULT_TABLE table=default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.182650 28456 consensus_meta.cc:287] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Time spent flushing consensus metadata: real 0.508s user 0.005s sys 0.000s
I20240503 14:12:11.183521 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 888636e9beca43edab96a8736751409e (DEFAULT_TABLE table=default.Table_d1c711b946024f31b19e3a3f2db88a0c [id=3854b08f7cd348c897aac4aacb7f5d06]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.185072 28427 leader_election.cc:341] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.192307 28462 tablet_service.cc:1467] Processing CreateTablet for tablet ac752d8ab4fd4a218b052f388f416494 (DEFAULT_TABLE table=default.Table_c7e5a76bf36545a5b53fc3d60512a521 [id=e6889b2bf6214cabba6669c414c22f7a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.192803 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 145562us
I20240503 14:12:11.195354 28604 tablet_bootstrap.cc:492] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:11.195593 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59945d011b64cba87c854e2ffde8226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.208642 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac752d8ab4fd4a218b052f388f416494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.212795 28604 ts_tablet_manager.cc:1367] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.576s user 0.004s sys 0.032s
I20240503 14:12:11.215277 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:11.215580 28212 raft_consensus.cc:2454] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:11.217032 28604 raft_consensus.cc:348] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.217463 29316 leader_election.cc:288] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.218756 29317 raft_consensus.cc:482] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.219508 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 167a926fa61747e2b4fcb52ba4eeea5b (DEFAULT_TABLE table=default.Table_961bb9bd6fec443589e047ead06c1526 [id=8362eec57bb34b0e976ebed080aa4e52]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.220944 28427 leader_election.cc:341] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 74df7781cc6a4dd6b6fd78a728ffd71b
I20240503 14:12:11.221140 28427 leader_election.cc:302] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:11.222071 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d77014c66624f958b38f7cc83644891" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:11.221318 29317 raft_consensus.cc:504] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:11.224308 28150 leader_election.cc:341] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.227545 29317 leader_election.cc:288] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.229570 29316 raft_consensus.cc:2726] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.244853 29316 consensus_queue.cc:1035] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.245083 28489 raft_consensus.cc:2454] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:11.245862 28661 ts_tablet_manager.cc:1888] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:11.246084 28661 log.cc:1196] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/40da5f5fff94462d8af740f3ee337e76
I20240503 14:12:11.246596 28661 ts_tablet_manager.cc:1909] T 40da5f5fff94462d8af740f3ee337e76 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:11.249574 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f223cbbcaa5402a888b084df0fe1c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.253298 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdc868ddf0a490292eae78ebc8c8ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.256255 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167a926fa61747e2b4fcb52ba4eeea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.259691 28604 raft_consensus.cc:374] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.259843 28604 raft_consensus.cc:729] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.260154 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b86451bc448428da7485e70e0d54b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.264288 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e220336186f648419b186feada6f0124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.268350 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c79ffee3954b5c870f4a8aa6379517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.274824 28653 tablet_replica.cc:330] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:11.275130 28653 raft_consensus.cc:2229] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:11.275355 28653 raft_consensus.cc:2258] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:11.276058 28653 ts_tablet_manager.cc:1875] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:11.284293 28604 consensus_queue.cc:260] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.285877 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888636e9beca43edab96a8736751409e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.292945 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd (DEFAULT_TABLE table=default.Table_8d1519e30ef54d12ab71e31903605531 [id=fa291ee5c8f54e89bcef6755f4ff3f0a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.293264 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 179102us
I20240503 14:12:11.295907 29317 consensus_queue.cc:1035] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.296336 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 with delete_type TABLET_DATA_DELETED (replaced by tablet d4c8a97076a44f9eaa3a15026f90a304) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.297662 28604 ts_tablet_manager.cc:1398] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.085s user 0.001s sys 0.001s
I20240503 14:12:11.309402 28604 tablet_bootstrap.cc:492] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:11.300839 28471 tablet_service.cc:1467] Processing CreateTablet for tablet aac51b83abe14ab5b1fb5d627233e58e (DEFAULT_TABLE table=default.Table_618386316f9b46f9a7770c3f998769f5 [id=ffcb298fb844406881d9a8812e24e896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.310027 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac51b83abe14ab5b1fb5d627233e58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.301287 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet e9609112657f427cbeeb3ef8f28cddf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.312053 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.312326 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.301949 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b52ea336524beea2f255c8720f7472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.302779 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c875ec8c5e49d1bb20a23dd7a21902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.303429 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.304283 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 646f83185d5241268d4126d10561e937 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.305778 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.306547 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f5103b31ed45b480252dcf94b72f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.297176 28427 leader_election.cc:302] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:11.332641 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.333355 29297 raft_consensus.cc:482] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.337764 29317 raft_consensus.cc:2781] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:11.337940 29317 raft_consensus.cc:482] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:11.338099 29317 raft_consensus.cc:3037] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.341228 28473 tablet_service.cc:1467] Processing CreateTablet for tablet d6f5103b31ed45b480252dcf94b72f9c (DEFAULT_TABLE table=default.Table_41a322f940d942adab752b8952a9f4a4 [id=18b53c1cdfb4468296f5bad79e15fb01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.356861 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.357435 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.348912 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 329) took 1799 ms. Trace:
I20240503 14:12:11.358841 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac7b6df20b14d1ead036cdc42174d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.361227 29266 raft_consensus.cc:504] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.362176 29266 leader_election.cc:288] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.362818 28661 rpcz_store.cc:276] 0503 14:12:09.547960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.550430 (+ 2470us) service_pool.cc:225] Handling call
0503 14:12:10.358094 (+807664us) ts_tablet_manager.cc:1120] Deleting tablet 40da5f5fff94462d8af740f3ee337e76
0503 14:12:10.358104 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.097950 (+739846us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.245694 (+147744us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.347316 (+101622us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58258,"tablet-delete.queue_time_us":807462}
I20240503 14:12:11.363176 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7295c2d3bd34e20a69986aadc9c7017" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:11.363955 28150 leader_election.cc:341] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.364168 28150 leader_election.cc:302] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.364672 29243 raft_consensus.cc:2726] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.364652 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet 74df7781cc6a4dd6b6fd78a728ffd71b with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8411f474c247c2abfd0bd9682fb8a2) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.364981 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d77014c66624f958b38f7cc83644891" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:11.365681 28150 leader_election.cc:341] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.365866 28150 leader_election.cc:302] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.366362 28614 tablet_bootstrap.cc:492] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:11.366400 29243 raft_consensus.cc:2726] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.368208 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 330) took 1806 ms. Trace:
I20240503 14:12:11.368418 28661 rpcz_store.cc:276] 0503 14:12:09.557312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.560214 (+ 2902us) service_pool.cc:225] Handling call
0503 14:12:11.363426 (+1803212us) ts_tablet_manager.cc:1120] Deleting tablet a8d2600031df4d77b9afe636a9b5ad40
0503 14:12:11.363440 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.363548 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1802900}
I20240503 14:12:11.369647 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet cae4969fc9474fdbaeae1b4d09e4875b (table default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f [id=837e8a35f3024ddcbe19763625d82a1f]) successfully deleted
I20240503 14:12:11.372148 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet bd76060daab84c2f9f6727b8a9bb3a49 (table default.Table_3668987e06c74fa6bc566d6e46f92879 [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
I20240503 14:12:11.378322 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 344) took 1569 ms. Trace:
I20240503 14:12:11.378593 28661 rpcz_store.cc:276] 0503 14:12:09.808915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.808965 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:11.376886 (+1567921us) ts_tablet_manager.cc:1120] Deleting tablet 779f373ef12a4da091309fcf247b5693
0503 14:12:11.376897 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.378215 (+ 1318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567631}
I20240503 14:12:11.386009 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 with delete_type TABLET_DATA_DELETED (replaced by tablet d4c8a97076a44f9eaa3a15026f90a304) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.386690 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet cae4969fc9474fdbaeae1b4d09e4875b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.387830 28290 leader_election.cc:302] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:11.388857 29297 raft_consensus.cc:504] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.379703 28470 tablet_service.cc:1467] Processing CreateTablet for tablet f8fe462c9ee54a27b9714f6457036d76 (DEFAULT_TABLE table=default.Table_285ab08f9ab94ca6a053a091945582ae [id=bd634bbb45704481a0476a7dfc8f0db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.380391 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7b805aee554f8fa4c198ce3409c6e6 (DEFAULT_TABLE table=default.Table_83e923042498440797a6b72edecd6277 [id=4df57b62a5b34ac182e799495a848041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.381031 28472 tablet_service.cc:1467] Processing CreateTablet for tablet c00dbc5a90674873b61b377e3e8d92fb (DEFAULT_TABLE table=default.Table_a186e8c3c3b440bba034806014b91bb6 [id=822d52647da14f238b5c547d4bbd9dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.384217 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f5103b31ed45b480252dcf94b72f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.391294 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7b805aee554f8fa4c198ce3409c6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.393283 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00dbc5a90674873b61b377e3e8d92fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.396994 28661 tablet_replica.cc:330] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:11.397310 28661 raft_consensus.cc:2229] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:11.397833 29243 raft_consensus.cc:482] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.398149 29243 raft_consensus.cc:504] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.398998 29243 leader_election.cc:288] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:11.403751 28661 raft_consensus.cc:2258] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:11.407353 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fe462c9ee54a27b9714f6457036d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.408118 28614 tablet_bootstrap.cc:654] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.389365 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.412472 29326 tablet.cc:1722] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:11.413131 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 82637.2us
I20240503 14:12:11.416317 29316 raft_consensus.cc:482] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.416641 29316 raft_consensus.cc:504] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.417477 29316 leader_election.cc:288] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.417712 29297 leader_election.cc:288] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:11.419275 29266 raft_consensus.cc:2781] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:11.419453 28603 ts_tablet_manager.cc:1888] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:11.420272 28465 tablet_service.cc:1467] Processing CreateTablet for tablet ea261d8d5d174bee804aa2e171f35008 (DEFAULT_TABLE table=default.Table_4f7b933efafa4a8080b087eff9410a90 [id=cf499dac577641bd8cd74d615076204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.420955 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:11.419888 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7b6df20b14d1ead036cdc42174d3a (DEFAULT_TABLE table=default.Table_cdd0b8d98dd949578e4a4525c2666b1c [id=0edef09212734f47be4ac77cca4fc186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.420884 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad1cd0e80cd4adbbbbcd60e335a5c49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:11.422040 28150 leader_election.cc:341] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.422457 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:11.422596 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9609112657f427cbeeb3ef8f28cddf0" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:11.422729 28212 raft_consensus.cc:2454] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:11.422785 29266 raft_consensus.cc:686] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:11.428375 29266 consensus_queue.cc:237] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.432041 28604 tablet_bootstrap.cc:654] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.419888 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad1cd0e80cd4adbbbbcd60e335a5c49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:11.434032 28603 log.cc:1196] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/f51834695b8b4a729a088da81eee2e47
I20240503 14:12:11.434680 28661 ts_tablet_manager.cc:1875] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:11.435037 29305 tablet.cc:1722] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 14:12:11.435277 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:11.436694 28150 leader_election.cc:341] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.436913 28150 leader_election.cc:302] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.437438 29243 raft_consensus.cc:2726] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.437960 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea261d8d5d174bee804aa2e171f35008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.438585 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 3f81bbdb2ee149c5a446c608034585fe (DEFAULT_TABLE table=default.Table_e02fae7462a5462eb76e4e39a374b6e1 [id=0a3c96abf47b4f7cb543422990ae8812]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.439137 28017 consensus_peers.cc:485] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 -> Peer a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259): Couldn't send request to peer a10ddca38c26462eb221d6026e88c6cc. Status: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:11.450261 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9609112657f427cbeeb3ef8f28cddf0" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:11.456274 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f81bbdb2ee149c5a446c608034585fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.464744 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 2b52ef7ecedf4273939406581d843980 (DEFAULT_TABLE table=default.Table_07fdeadf61a045d8aeb377c2a3c4c1af [id=be2dbf4734c74e93ad430d203979e633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.465031 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:11.472259 28603 ts_tablet_manager.cc:1909] T f51834695b8b4a729a088da81eee2e47 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:11.476225 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 347) took 1572 ms. Trace:
I20240503 14:12:11.476435 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 08cc84cf703048148abfe03b7dee12ce (DEFAULT_TABLE table=default.Table_20718ac286c9405b871c43b98ac94f8d [id=51930b6b93bd406581a479188de3423d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.474269 29317 raft_consensus.cc:504] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.479249 28603 rpcz_store.cc:276] 0503 14:12:09.900290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.900353 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:10.645770 (+745417us) ts_tablet_manager.cc:1120] Deleting tablet f51834695b8b4a729a088da81eee2e47
0503 14:12:10.645779 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:10.812469 (+166690us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.385536 (+573067us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.472713 (+ 87177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":744416}
I20240503 14:12:11.480146 28455 tablet_service.cc:1467] Processing CreateTablet for tablet a26e8389639748ae99832d7a7a9d5af3 (DEFAULT_TABLE table=default.Table_5bbe821329104389be37f7890f051041 [id=78e7487e239c4115ae6f3a3a45ca8b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.480296 28603 tablet_replica.cc:330] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:11.480989 28603 raft_consensus.cc:2229] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:11.481205 28603 raft_consensus.cc:2258] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:11.482463 28614 tablet_bootstrap.cc:492] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:11.482434 28603 ts_tablet_manager.cc:1875] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:11.482784 28614 ts_tablet_manager.cc:1367] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.464s user 0.002s sys 0.007s
I20240503 14:12:11.483397 28462 tablet_service.cc:1467] Processing CreateTablet for tablet a1ebec4622ef4950bd1c70451958c59b (DEFAULT_TABLE table=default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.484151 28457 tablet_service.cc:1467] Processing CreateTablet for tablet e45df464f6954a56bec8b107b350320c (DEFAULT_TABLE table=default.Table_d5808426f2954b4981e680e8a56f5a5b [id=beabb84faa9f4d34a0293ea09c93e19a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.484675 28614 raft_consensus.cc:348] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.485744 29317 leader_election.cc:288] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:11.485720 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 62553.2us
I20240503 14:12:11.491050 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet f51834695b8b4a729a088da81eee2e47 (table default.Table_b0e228f901e4427ca9bbfc14b02c1f7f [id=743c045f2e104da8b09fd13245037eb6]) successfully deleted
I20240503 14:12:11.491547 28290 leader_election.cc:302] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:11.493121 28290 leader_election.cc:302] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.485116 28614 raft_consensus.cc:374] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.493903 29297 raft_consensus.cc:2781] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:11.494355 29297 raft_consensus.cc:482] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:11.494531 29297 raft_consensus.cc:3037] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.496054 29330 raft_consensus.cc:2726] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:11.497328 28600 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.233s sys 0.225s
W20240503 14:12:11.497493 28600 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.564s user 0.233s sys 0.225s
I20240503 14:12:11.499971 28600 tablet_bootstrap.cc:492] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:11.500337 28600 ts_tablet_manager.cc:1367] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.574s user 0.236s sys 0.226s
I20240503 14:12:11.501209 29266 raft_consensus.cc:482] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.501806 28600 raft_consensus.cc:348] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.502506 28600 raft_consensus.cc:374] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.502723 28600 raft_consensus.cc:729] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.503090 28600 consensus_queue.cc:260] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.504280 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet dbcc6cef7b7d464a9fbbae61caf0437d
W20240503 14:12:11.506415 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea261d8d5d174bee804aa2e171f35008 consensus state has no leader
I20240503 14:12:11.506719 28600 ts_tablet_manager.cc:1398] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:11.507102 28600 tablet_bootstrap.cc:492] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:11.507112 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 consensus state has no leader
I20240503 14:12:11.508852 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8410e8b6b511403ea609b1353855937b
I20240503 14:12:11.511850 28600 tablet_bootstrap.cc:654] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.515566 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.518446 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.519583 28614 raft_consensus.cc:729] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.522894 29266 raft_consensus.cc:504] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.523463 28600 tablet_bootstrap.cc:492] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:11.523600 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.523767 28600 ts_tablet_manager.cc:1367] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20240503 14:12:11.523991 29266 leader_election.cc:288] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.525135 28600 raft_consensus.cc:348] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.525615 28600 raft_consensus.cc:374] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.525794 28600 raft_consensus.cc:729] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.526232 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410e8b6b511403ea609b1353855937b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.526450 28600 consensus_queue.cc:260] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.526775 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.530558 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.526268 28614 consensus_queue.cc:260] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.535084 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 with delete_type TABLET_DATA_DELETED (replaced by tablet d4c8a97076a44f9eaa3a15026f90a304) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.544761 28491 raft_consensus.cc:2454] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:11.544826 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410e8b6b511403ea609b1353855937b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.546996 28614 ts_tablet_manager.cc:1398] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.063s user 0.001s sys 0.003s
I20240503 14:12:11.550398 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c8a1b23185cf4c51ac94cb268e9e2222"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:11.551314 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8a1b23185cf4c51ac94cb268e9e2222 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.553886 29297 raft_consensus.cc:504] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.554764 29297 leader_election.cc:288] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.555730 28614 tablet_bootstrap.cc:492] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:11.557704 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac7b6df20b14d1ead036cdc42174d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.557801 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ab664e165be64811ae894db7bab917d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:11.558580 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab664e165be64811ae894db7bab917d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.558388 29324 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:11.560492 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:11.560768 28491 raft_consensus.cc:3037] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.562425 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7b4a078c66b0477180bed5b7b139de58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:11.563346 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e145076fa64d4661bf830b556d2d2351" } new_table_name: "default.Table_bc29cf4457ea41e5a9d22f6d8c120f36" modify_external_catalogs: true
I20240503 14:12:11.565003 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45df464f6954a56bec8b107b350320c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:11.565444 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b4a078c66b0477180bed5b7b139de58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.565681 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebec4622ef4950bd1c70451958c59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.566480 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 004d556f3ed44a7db1a52d205ecfd90b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:11.570876 29326 tablet.cc:1722] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 14:12:11.571090 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76]). Expected version 2 got 0
W20240503 14:12:11.572182 28290 leader_election.cc:341] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: cae4969fc9474fdbaeae1b4d09e4875b
I20240503 14:12:11.575260 28614 tablet_bootstrap.cc:654] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.557574 29243 raft_consensus.cc:482] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.580037 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_44fe718f4ee1446bba6f7cd746b0154a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:11.580411 29243 raft_consensus.cc:504] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:11.580893 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44fe718f4ee1446bba6f7cd746b0154a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.581276 29243 leader_election.cc:288] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:11.561872 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:11.581632 28212 raft_consensus.cc:3037] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:11.582491 28290 leader_election.cc:341] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: cae4969fc9474fdbaeae1b4d09e4875b
I20240503 14:12:11.583029 28290 leader_election.cc:302] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.583184 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cdd0b8d98dd949578e4a4525c2666b1c" } new_table_name: "default.Table_556c4b7d2cad40c3b8520f4269675dc8" modify_external_catalogs: true
I20240503 14:12:11.583828 29297 raft_consensus.cc:2726] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.577142 29281 raft_consensus.cc:482] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.584264 29281 raft_consensus.cc:504] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.585086 29281 leader_election.cc:288] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:11.586530 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0982b99a02ee48f2b6379e7be56c4c02" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:11.586616 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:11.588456 28150 leader_election.cc:341] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:11.588780 28150 leader_election.cc:341] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.588868 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0982b99a02ee48f2b6379e7be56c4c02" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:11.589778 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:11.592281 28150 leader_election.cc:341] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.592479 28150 leader_election.cc:302] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:11.592972 28150 leader_election.cc:341] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.593158 28150 leader_election.cc:302] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.593472 28459 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51314 (request call id 385) took 1014 ms. Trace:
I20240503 14:12:11.593655 28459 rpcz_store.cc:276] 0503 14:12:10.579189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.801788 (+222599us) service_pool.cc:225] Handling call
0503 14:12:10.885545 (+ 83757us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 14:12:10.885579 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0503 14:12:11.255429 (+369850us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.445939 (+190510us) tablet_replica.cc:205] Creating consensus instance
0503 14:12:11.593402 (+147463us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3045}
I20240503 14:12:11.593870 29243 raft_consensus.cc:2726] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.594116 29243 raft_consensus.cc:2726] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.595041 28600 ts_tablet_manager.cc:1398] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.071s user 0.003s sys 0.000s
I20240503 14:12:11.604192 28600 tablet_bootstrap.cc:492] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:11.609630 28600 tablet_bootstrap.cc:654] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.613415 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba (table default.Table_a888de5f6a40467db886c3141d2a87cc [id=caca4dbaa701453794ac56475f46704e]). Expected version 2 got 0
I20240503 14:12:11.614562 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 004d556f3ed44a7db1a52d205ecfd90b
I20240503 14:12:11.620385 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.622084 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6a58db50f849a8be729c10dbe03e76 (DEFAULT_TABLE table=default.Table_6dfb1a277c824cbfad52ce51179289a6 [id=6199baccf6a9435f8d53d80f58531766]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.622690 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.623247 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6a58db50f849a8be729c10dbe03e76 (DEFAULT_TABLE table=default.Table_6dfb1a277c824cbfad52ce51179289a6 [id=6199baccf6a9435f8d53d80f58531766]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.623293 28180 tablet_service.cc:1467] Processing CreateTablet for tablet a2369dd6d16e4b9681108f2c8574f658 (DEFAULT_TABLE table=default.Table_87bf7222c4444f8f90511d56ac011976 [id=cdef8f6c86bd445d8419ded630b5610c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.623919 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 (DEFAULT_TABLE table=default.Table_f2b1cc65e0f5452fbc24389d16be4eb5 [id=758923d87c0943429c4bb0287921f25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.624294 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d77014c66624f958b38f7cc83644891 with delete_type TABLET_DATA_DELETED (replaced by tablet 20b176a071ba4f3fbff2a64d6e46027a) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.624353 28325 tablet_service.cc:1467] Processing CreateTablet for tablet a2369dd6d16e4b9681108f2c8574f658 (DEFAULT_TABLE table=default.Table_87bf7222c4444f8f90511d56ac011976 [id=cdef8f6c86bd445d8419ded630b5610c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.625358 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 3f1aa939aced4556ae0f0de0e62d1f97 (table default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb]). Expected version 4 got 0
I20240503 14:12:11.625651 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 (DEFAULT_TABLE table=default.Table_f2b1cc65e0f5452fbc24389d16be4eb5 [id=758923d87c0943429c4bb0287921f25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.625615 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]). Expected version 2 got 0
I20240503 14:12:11.626053 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:11.629173 28614 tablet_bootstrap.cc:492] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:11.629437 28614 ts_tablet_manager.cc:1367] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.000s
I20240503 14:12:11.630438 28614 raft_consensus.cc:348] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.630748 28614 raft_consensus.cc:374] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.630859 28614 raft_consensus.cc:729] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.631182 28614 consensus_queue.cc:260] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.632122 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e446d24cec95467d98278762338e11d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:11.632871 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e446d24cec95467d98278762338e11d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.635205 28351 raft_consensus.cc:3037] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.635543 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 6b94092725fa4956a4c00e3685f9aeaf (DEFAULT_TABLE table=default.Table_4fc8bf5e30814b5eb9e5ea28422b1802 [id=cc4a64a0a7ed44049d6b34e343ccf051]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.636312 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 0f75e102dd4249c3800e1fcb4339acf3 (DEFAULT_TABLE table=default.Table_cb1bb0240e554fb2a420db2ec7d00e68 [id=18f3634a9252400cbedc9a5cce56388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.637140 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c383582e14364f50a46da3692bedd880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:11.637473 28327 tablet_service.cc:1467] Processing CreateTablet for tablet b24ab3a5bba04a5092225a981f292222 (DEFAULT_TABLE table=default.Table_34588b8057d642378dc78cac201ee75f [id=294277efc25b401291a90d0fa70c90a9]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:11.637799 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c383582e14364f50a46da3692bedd880 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.638449 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 788e4bfa1e414630939bc19da1ede3fb (DEFAULT_TABLE table=default.Table_0f214d103abd4357829af22666b5f9a7 [id=ddef27f860534921adab476e0f391782]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.639307 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4fa4898a059c4416b632f8de37cd94d1" } new_table_name: "default.Table_272609e488f340d29926451ba0376de0" modify_external_catalogs: true
I20240503 14:12:11.639431 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 6b94092725fa4956a4c00e3685f9aeaf (DEFAULT_TABLE table=default.Table_4fc8bf5e30814b5eb9e5ea28422b1802 [id=cc4a64a0a7ed44049d6b34e343ccf051]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.640465 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 0f75e102dd4249c3800e1fcb4339acf3 (DEFAULT_TABLE table=default.Table_cb1bb0240e554fb2a420db2ec7d00e68 [id=18f3634a9252400cbedc9a5cce56388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.641163 28614 ts_tablet_manager.cc:1398] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.012s user 0.003s sys 0.001s
I20240503 14:12:11.642846 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_78f62674ace541fabf9047f21a7cadbc" } modify_external_catalogs: true
I20240503 14:12:11.643126 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_78f62674ace541fabf9047f21a7cadbc" } modify_external_catalogs: true
I20240503 14:12:11.644448 28182 tablet_service.cc:1467] Processing CreateTablet for tablet b24ab3a5bba04a5092225a981f292222 (DEFAULT_TABLE table=default.Table_34588b8057d642378dc78cac201ee75f [id=294277efc25b401291a90d0fa70c90a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.645062 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 788e4bfa1e414630939bc19da1ede3fb (DEFAULT_TABLE table=default.Table_0f214d103abd4357829af22666b5f9a7 [id=ddef27f860534921adab476e0f391782]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.647469 28614 tablet_bootstrap.cc:492] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:11.650988 28212 raft_consensus.cc:2454] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:11.652567 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6a58db50f849a8be729c10dbe03e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.653254 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b94092725fa4956a4c00e3685f9aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.654924 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24ab3a5bba04a5092225a981f292222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.655615 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a9d58fb30d4b60ae0a44ec217fea11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.656266 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788e4bfa1e414630939bc19da1ede3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.656857 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2369dd6d16e4b9681108f2c8574f658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.660054 28290 leader_election.cc:302] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:11.660372 28603 ts_tablet_manager.cc:1888] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:11.660601 28603 log.cc:1196] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/1693dd36a3d84d88a49690c537590083
I20240503 14:12:11.660655 29297 raft_consensus.cc:2781] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:11.660843 29297 raft_consensus.cc:686] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:11.660996 28600 tablet_bootstrap.cc:492] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:11.661149 28603 ts_tablet_manager.cc:1909] T 1693dd36a3d84d88a49690c537590083 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:11.661196 28600 ts_tablet_manager.cc:1367] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.003s
I20240503 14:12:11.661283 29297 consensus_queue.cc:237] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.662293 28600 raft_consensus.cc:348] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.662549 28600 raft_consensus.cc:374] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.662681 28600 raft_consensus.cc:729] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.663019 28600 consensus_queue.cc:260] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.664659 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f75e102dd4249c3800e1fcb4339acf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.672349 29330 raft_consensus.cc:482] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.672631 29330 raft_consensus.cc:504] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.673794 29330 leader_election.cc:288] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.675243 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc230b489ab8455c8cb2eba31084fddb" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:11.675962 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc230b489ab8455c8cb2eba31084fddb" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:11.677454 28614 tablet_bootstrap.cc:654] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.680358 28491 raft_consensus.cc:2454] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:11.685146 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:11.685483 28212 raft_consensus.cc:2379] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:11.685673 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cbd1326869434594e126421ad132d8" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:11.689291 28600 ts_tablet_manager.cc:1398] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.028s user 0.001s sys 0.002s
I20240503 14:12:11.691344 28600 tablet_bootstrap.cc:492] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:11.696229 29305 tablet.cc:1717] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 2, tablet already version 2
I20240503 14:12:11.696449 28290 leader_election.cc:302] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.696684 28653 ts_tablet_manager.cc:1888] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:11.696892 29330 raft_consensus.cc:2726] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.696920 28653 log.cc:1196] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/265dc5f2630e453ea71abddb1a9753f3
I20240503 14:12:11.697328 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24ab3a5bba04a5092225a981f292222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.698105 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788e4bfa1e414630939bc19da1ede3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.698318 29334 tablet.cc:1717] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 2, tablet already version 2
I20240503 14:12:11.698760 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a9d58fb30d4b60ae0a44ec217fea11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.696697 29305 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58172 (request call id 400) took 1159 ms. Trace:
I20240503 14:12:11.699373 29305 rpcz_store.cc:276] 0503 14:12:10.537023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.537085 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:11.696620 (+1159535us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:10.537574 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:10.537681 (+ 107us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:10.944710 (+407029us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:10.944725 (+ 15us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:10.944801 (+ 76us) alter_schema_op.cc:119] START. Timestamp: 1714745530944757
0503 14:12:10.944805 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:10.945051 (+ 246us) log.cc:842] Serialized 173 byte log entry
0503 14:12:11.695872 (+750821us) op_driver.cc:444] REPLICATION: finished
0503 14:12:11.696042 (+ 170us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:11.696533 (+ 491us) log.cc:842] Serialized 94 byte log entry
0503 14:12:11.696585 (+ 52us) alter_schema_op.cc:65] Released schema lock
0503 14:12:11.696588 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":42319,"prepare.run_wall_time_us":407581,"raft.queue_time_us":54,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":751028}]]}
I20240503 14:12:11.700284 29305 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:11.701175 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2369dd6d16e4b9681108f2c8574f658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.701867 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f75e102dd4249c3800e1fcb4339acf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.702613 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b94092725fa4956a4c00e3685f9aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.703321 29330 raft_consensus.cc:482] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.703578 29330 raft_consensus.cc:504] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.703810 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6a58db50f849a8be729c10dbe03e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.704453 29330 leader_election.cc:288] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.707271 28653 ts_tablet_manager.cc:1909] T 265dc5f2630e453ea71abddb1a9753f3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:11.711436 28290 leader_election.cc:341] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 74df7781cc6a4dd6b6fd78a728ffd71b
I20240503 14:12:11.711755 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74df7781cc6a4dd6b6fd78a728ffd71b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:11.713680 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32b6995818041778617a0d6bf936c8c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:11.714085 28356 raft_consensus.cc:2454] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:11.714562 28489 raft_consensus.cc:2454] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:11.715396 28290 leader_election.cc:302] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:11.715876 29330 raft_consensus.cc:2781] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:11.716009 29330 raft_consensus.cc:482] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:11.716285 29330 raft_consensus.cc:3037] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.717789 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:11.718453 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7945118bb17b431dac8670007d2d0a18 consensus state has no leader
I20240503 14:12:11.719507 28614 tablet_bootstrap.cc:492] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:11.725638 29317 consensus_queue.cc:1035] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.727700 29317 consensus_queue.cc:1035] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.728075 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 40da5f5fff94462d8af740f3ee337e76 (table default.Table_63619685889a4263ae065b254160cf30 [id=6a81b734738e4d998215938d5ca5dad7]) successfully deleted
W20240503 14:12:11.728493 28427 leader_election.cc:341] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:11.728564 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a8d2600031df4d77b9afe636a9b5ad40 because the tablet was not found. No further retry: Not found: Tablet not found: a8d2600031df4d77b9afe636a9b5ad40
W20240503 14:12:11.728863 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 779f373ef12a4da091309fcf247b5693 because the tablet was not found. No further retry: Not found: Tablet not found: 779f373ef12a4da091309fcf247b5693
I20240503 14:12:11.730548 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 4b5992e28f1745678618230c183609e4 (table default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951]). Expected version 2 got 0
I20240503 14:12:11.731009 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]). Expected version 2 got 0
I20240503 14:12:11.737509 28356 raft_consensus.cc:3037] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.739012 28604 tablet_bootstrap.cc:492] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:11.739221 28604 ts_tablet_manager.cc:1367] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.430s user 0.000s sys 0.009s
I20240503 14:12:11.740008 29281 raft_consensus.cc:482] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.741291 28427 leader_election.cc:302] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:11.742789 29325 raft_consensus.cc:2781] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:11.742933 29325 raft_consensus.cc:482] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:11.743077 29325 raft_consensus.cc:3037] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.746567 29325 raft_consensus.cc:504] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.747431 29325 leader_election.cc:288] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:11.748750 28427 leader_election.cc:341] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.756104 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410e8b6b511403ea609b1353855937b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.756824 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6a58db50f849a8be729c10dbe03e76 (DEFAULT_TABLE table=default.Table_6dfb1a277c824cbfad52ce51179289a6 [id=6199baccf6a9435f8d53d80f58531766]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.766531 28469 tablet_service.cc:1467] Processing CreateTablet for tablet a2369dd6d16e4b9681108f2c8574f658 (DEFAULT_TABLE table=default.Table_87bf7222c4444f8f90511d56ac011976 [id=cdef8f6c86bd445d8419ded630b5610c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.767170 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2369dd6d16e4b9681108f2c8574f658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.768177 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 (DEFAULT_TABLE table=default.Table_f2b1cc65e0f5452fbc24389d16be4eb5 [id=758923d87c0943429c4bb0287921f25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.768656 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a9d58fb30d4b60ae0a44ec217fea11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.770681 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:11.777532 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cc84cf703048148abfe03b7dee12ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.778347 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26e8389639748ae99832d7a7a9d5af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.779304 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b52ef7ecedf4273939406581d843980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.782406 29281 raft_consensus.cc:504] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.784577 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 382) took 1470 ms. Trace:
I20240503 14:12:11.784806 28653 rpcz_store.cc:276] 0503 14:12:10.312138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.335762 (+ 23624us) service_pool.cc:225] Handling call
0503 14:12:11.274689 (+938927us) ts_tablet_manager.cc:1120] Deleting tablet 265dc5f2630e453ea71abddb1a9753f3
0503 14:12:11.274704 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.465539 (+190835us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.696500 (+230961us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.782898 (+ 86398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":938215}
I20240503 14:12:11.785600 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d556f3ed44a7db1a52d205ecfd90b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:11.792546 28600 tablet_bootstrap.cc:654] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.793538 29281 leader_election.cc:288] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:11.792580 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 384) took 1454 ms. Trace:
W20240503 14:12:11.792551 28150 leader_election.cc:341] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.794790 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 265dc5f2630e453ea71abddb1a9753f3 (table default.Table_cba5fd0a90c8433980631ba7e2048d8b [id=3368a0430dc94c4db0ae6260f5a0acf0]) successfully deleted
I20240503 14:12:11.796391 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32b6995818041778617a0d6bf936c8c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:11.796736 28350 raft_consensus.cc:3037] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.799083 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32b6995818041778617a0d6bf936c8c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:11.799434 28216 raft_consensus.cc:2454] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:11.800276 28653 rpcz_store.cc:276] 0503 14:12:10.330600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.337836 (+ 7236us) service_pool.cc:225] Handling call
0503 14:12:11.785181 (+1447345us) ts_tablet_manager.cc:1120] Deleting tablet 20b176a071ba4f3fbff2a64d6e46027a
0503 14:12:11.785194 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.785353 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1443313}
I20240503 14:12:11.801641 28469 tablet_service.cc:1467] Processing CreateTablet for tablet b24ab3a5bba04a5092225a981f292222 (DEFAULT_TABLE table=default.Table_34588b8057d642378dc78cac201ee75f [id=294277efc25b401291a90d0fa70c90a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.804436 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_472814febc0548599d68e796d57f48a1" } modify_external_catalogs: true
I20240503 14:12:11.804793 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_472814febc0548599d68e796d57f48a1" } modify_external_catalogs: true
I20240503 14:12:11.802515 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 788e4bfa1e414630939bc19da1ede3fb (DEFAULT_TABLE table=default.Table_0f214d103abd4357829af22666b5f9a7 [id=ddef27f860534921adab476e0f391782]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.802960 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6a58db50f849a8be729c10dbe03e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.811682 28653 tablet_replica.cc:330] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:11.812008 28653 raft_consensus.cc:2229] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:11.812333 28653 raft_consensus.cc:2258] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:11.813019 28653 ts_tablet_manager.cc:1875] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:11.813793 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 6b94092725fa4956a4c00e3685f9aeaf (DEFAULT_TABLE table=default.Table_4fc8bf5e30814b5eb9e5ea28422b1802 [id=cc4a64a0a7ed44049d6b34e343ccf051]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.811057 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:11.814203 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788e4bfa1e414630939bc19da1ede3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.815747 28215 raft_consensus.cc:3037] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.817652 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32b6995818041778617a0d6bf936c8c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:11.817975 28213 raft_consensus.cc:3037] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:11.818146 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d556f3ed44a7db1a52d205ecfd90b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:11.818910 28150 leader_election.cc:341] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.819108 28150 leader_election.cc:302] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.819841 29281 raft_consensus.cc:2726] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:11.824025 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 20b176a071ba4f3fbff2a64d6e46027a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20b176a071ba4f3fbff2a64d6e46027a already in progress: creating tablet
I20240503 14:12:11.826740 28427 leader_election.cc:302] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:11.822870 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 0f75e102dd4249c3800e1fcb4339acf3 (DEFAULT_TABLE table=default.Table_cb1bb0240e554fb2a420db2ec7d00e68 [id=18f3634a9252400cbedc9a5cce56388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:11.829593 28661 ts_tablet_manager.cc:1888] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:11.829949 28661 log.cc:1196] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6dab0296a0894a63afc16ed466c591d5
I20240503 14:12:11.838516 28604 raft_consensus.cc:348] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.839015 28604 raft_consensus.cc:374] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.839149 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet 0af0342e0e0b47499a9322b5b2cb3d03 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:11.839270 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet ffbd3291c56b4c13813c80d6775afe6f from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:11.839246 28604 raft_consensus.cc:729] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.839679 28604 consensus_queue.cc:260] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.840011 29334 tablet.cc:1722] T 4b5992e28f1745678618230c183609e4 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:11.840109 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1c2668b8ddb741298f29976408a1caee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:11.840026 29325 raft_consensus.cc:2726] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:11.840749 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c2668b8ddb741298f29976408a1caee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.842674 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24ab3a5bba04a5092225a981f292222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.844976 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b94092725fa4956a4c00e3685f9aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.846963 28350 raft_consensus.cc:2454] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:11.850373 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f75e102dd4249c3800e1fcb4339acf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:11.862874 28213 raft_consensus.cc:2454] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:11.863622 28427 leader_election.cc:302] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:11.874341 28661 ts_tablet_manager.cc:1909] T 6dab0296a0894a63afc16ed466c591d5 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:11.882719 29325 raft_consensus.cc:2781] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:11.890208 29325 raft_consensus.cc:686] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:11.892316 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 345) took 2082 ms. Trace:
I20240503 14:12:11.894073 28614 ts_tablet_manager.cc:1367] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.001s
I20240503 14:12:11.896708 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 346) took 1862 ms. Trace:
I20240503 14:12:11.892120 29358 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:11.903810 28603 rpcz_store.cc:276] 0503 14:12:09.896821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.897088 (+ 267us) service_pool.cc:225] Handling call
0503 14:12:11.479779 (+1582691us) ts_tablet_manager.cc:1120] Deleting tablet 1693dd36a3d84d88a49690c537590083
0503 14:12:11.479792 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.548430 (+ 68638us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.652061 (+103631us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.752814 (+100753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577738}
I20240503 14:12:11.900847 28614 raft_consensus.cc:348] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.906787 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 349) took 1957 ms. Trace:
I20240503 14:12:11.906896 28614 raft_consensus.cc:374] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.906942 28603 rpcz_store.cc:276] 0503 14:12:09.948322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.948381 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:11.904280 (+1955899us) ts_tablet_manager.cc:1120] Deleting tablet e0175b38c1e0464a9a6208cfaf68ca0d
0503 14:12:11.904293 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.905702 (+ 1409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1955624}
I20240503 14:12:11.907480 29281 raft_consensus.cc:482] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.907706 29281 raft_consensus.cc:504] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.905165 28604 ts_tablet_manager.cc:1398] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.138s user 0.002s sys 0.002s
I20240503 14:12:11.908310 29316 raft_consensus.cc:482] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:11.908567 29281 leader_election.cc:288] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:11.908967 28614 raft_consensus.cc:729] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.909310 28614 consensus_queue.cc:260] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.890731 29356 tablet.cc:1722] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20240503 14:12:11.912151 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e0175b38c1e0464a9a6208cfaf68ca0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0175b38c1e0464a9a6208cfaf68ca0d already in progress: creating tablet
I20240503 14:12:11.914424 29316 raft_consensus.cc:504] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.916656 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fdf88fefa4ff7939f3ca05a82d7ba" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:11.917528 28150 leader_election.cc:341] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.919306 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 11ac912a3bd64cdc9082596e09c72e13
W20240503 14:12:11.919358 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ac7b6df20b14d1ead036cdc42174d3a consensus state has no leader
I20240503 14:12:11.893020 29360 tablet.cc:1722] T 4b5992e28f1745678618230c183609e4 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:11.921590 28604 tablet_bootstrap.cc:492] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:11.921926 29325 consensus_queue.cc:237] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.922295 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7b9e454770964e4594e5d043b06c2e78" } modify_external_catalogs: true
I20240503 14:12:11.922438 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_34588b8057d642378dc78cac201ee75f" } new_table_name: "default.Table_21dbc608308a4881af80133c4e9e04fd" modify_external_catalogs: true
I20240503 14:12:11.922677 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7b9e454770964e4594e5d043b06c2e78" } modify_external_catalogs: true
I20240503 14:12:11.924211 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c12ba1b49a3f4f509ff427bbeb4723af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:11.924885 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c12ba1b49a3f4f509ff427bbeb4723af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:11.929898 29316 leader_election.cc:288] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:11.929994 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fdf88fefa4ff7939f3ca05a82d7ba" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:11.932994 28150 leader_election.cc:341] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:11.933391 28150 leader_election.cc:302] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:11.933920 29281 raft_consensus.cc:2726] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:11.934383 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:11.935920 28600 tablet_bootstrap.cc:492] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:11.936228 28600 ts_tablet_manager.cc:1367] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.247s user 0.001s sys 0.007s
I20240503 14:12:11.938069 28600 raft_consensus.cc:348] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.938501 28600 raft_consensus.cc:374] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:11.937712 28661 rpcz_store.cc:276] 0503 14:12:09.809411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.809452 (+ 41us) service_pool.cc:225] Handling call
0503 14:12:11.396706 (+1587254us) ts_tablet_manager.cc:1120] Deleting tablet 6dab0296a0894a63afc16ed466c591d5
0503 14:12:11.396721 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.692320 (+295599us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.801291 (+108971us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.892236 (+ 90945us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587073}
I20240503 14:12:11.938747 28600 raft_consensus.cc:729] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:11.939200 28600 consensus_queue.cc:260] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:11.939440 28661 tablet_replica.cc:330] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:11.940896 28661 raft_consensus.cc:2229] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:11.942059 28661 raft_consensus.cc:2258] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:11.946847 28600 ts_tablet_manager.cc:1398] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:12:11.947849 28661 ts_tablet_manager.cc:1875] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:11.948376 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 348) took 1963 ms. Trace:
I20240503 14:12:11.953193 29324 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51314 (request call id 348) took 2077 ms. Trace:
I20240503 14:12:11.953392 29324 rpcz_store.cc:276] 0503 14:12:09.876101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.876163 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:11.953112 (+2076949us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:09.903855 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:09.904046 (+ 191us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:09.904057 (+ 11us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:09.904065 (+ 8us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:09.904141 (+ 76us) alter_schema_op.cc:119] START. Timestamp: 1714745529904096
0503 14:12:09.904145 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:09.904410 (+ 265us) log.cc:842] Serialized 173 byte log entry
0503 14:12:11.294556 (+1390146us) op_driver.cc:444] REPLICATION: finished
0503 14:12:11.475737 (+181181us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:11.559687 (+ 83950us) log.cc:842] Serialized 5 byte log entry
0503 14:12:11.952705 (+393018us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:11.953040 (+ 335us) log.cc:842] Serialized 14 byte log entry
0503 14:12:11.953080 (+ 40us) alter_schema_op.cc:65] Released schema lock
0503 14:12:11.953083 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":167682,"prepare.queue_time_us":12043,"prepare.run_cpu_time_us":674,"prepare.run_wall_time_us":1099,"raft.queue_time_us":148,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":325,"replication_time_us":1390375,"thread_start_us":256671,"threads_started":3,"wal-append.queue_time_us":125713}]]}
I20240503 14:12:11.954658 28614 ts_tablet_manager.cc:1398] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20240503 14:12:11.955984 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:11.955312 28603 rpcz_store.cc:276] 0503 14:12:09.948168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.952485 (+ 4317us) service_pool.cc:225] Handling call
0503 14:12:11.907225 (+1954740us) ts_tablet_manager.cc:1120] Deleting tablet 0b139ab0b5864024ac48215d46b0a7bd
0503 14:12:11.907236 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.911827 (+ 4591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1954302}
I20240503 14:12:11.959450 28614 tablet_bootstrap.cc:492] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:11.958070 28604 tablet_bootstrap.cc:654] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.949882 28215 raft_consensus.cc:2454] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:11.971701 28600 tablet_bootstrap.cc:492] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:11.973502 28603 tablet_replica.cc:330] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:11.973865 28600 tablet_bootstrap.cc:654] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:11.975731 28427 leader_election.cc:302] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:11.977706 29317 raft_consensus.cc:2781] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:11.978222 29317 raft_consensus.cc:686] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:11.978402 28603 raft_consensus.cc:2229] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:11.978768 29317 consensus_queue.cc:237] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:11.980293 28603 raft_consensus.cc:2258] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:11.981063 28603 ts_tablet_manager.cc:1875] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:11.982206 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0b139ab0b5864024ac48215d46b0a7bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b139ab0b5864024ac48215d46b0a7bd already in progress: creating tablet
I20240503 14:12:11.992427 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129717ef137c41c5a08d401c64164187" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:11.992727 28215 raft_consensus.cc:2454] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:11.992720 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129717ef137c41c5a08d401c64164187" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:11.998068 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02dd4c914a4846bab3a00408247ec64a consensus state has no leader
I20240503 14:12:12.000770 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2" } new_table_name: "default.Table_fc189ad8c19041f7814fac6de9f482cb" modify_external_catalogs: true
I20240503 14:12:12.002732 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2907862ac0af4c3692460a0071e8bce3" } modify_external_catalogs: true
I20240503 14:12:12.002969 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2907862ac0af4c3692460a0071e8bce3" } modify_external_catalogs: true
I20240503 14:12:12.005865 29305 tablet.cc:1722] T 4b5992e28f1745678618230c183609e4 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:12.004531 29330 raft_consensus.cc:504] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.008885 28029 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76] alter complete (version 2)
I20240503 14:12:12.010370 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.015681 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet e9609112657f427cbeeb3ef8f28cddf0 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:12.015733 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74df7781cc6a4dd6b6fd78a728ffd71b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:12.015213 28427 leader_election.cc:302] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:12.016443 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ff4d28d3c2ea472191ad21355248a2fc
I20240503 14:12:12.017792 29316 raft_consensus.cc:2781] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:12.017954 29316 raft_consensus.cc:482] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:12.018065 29316 raft_consensus.cc:3037] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:12.019433 28653 ts_tablet_manager.cc:1888] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.019608 28653 log.cc:1196] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0982b99a02ee48f2b6379e7be56c4c02
I20240503 14:12:12.020037 28653 ts_tablet_manager.cc:1909] T 0982b99a02ee48f2b6379e7be56c4c02 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:12.020939 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 383) took 1690 ms. Trace:
I20240503 14:12:12.021086 28653 rpcz_store.cc:276] 0503 14:12:10.330441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.338063 (+ 7622us) service_pool.cc:225] Handling call
0503 14:12:11.800601 (+1462538us) ts_tablet_manager.cc:1120] Deleting tablet 0982b99a02ee48f2b6379e7be56c4c02
0503 14:12:11.800614 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:11.954653 (+154039us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.019228 (+ 64575us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.020895 (+ 1667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1458630}
I20240503 14:12:12.021454 28653 tablet_replica.cc:330] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:12.021452 29316 raft_consensus.cc:504] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.021714 28653 raft_consensus.cc:2229] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.021914 28653 raft_consensus.cc:2258] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.022521 29316 leader_election.cc:288] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.022584 28653 ts_tablet_manager.cc:1875] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:12.027050 29359 log.cc:924] Time spent T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20240503 14:12:12.027822 29334 tablet.cc:1717] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 4, tablet already version 4
I20240503 14:12:12.028214 29334 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51314 (request call id 367) took 1619 ms. Trace:
I20240503 14:12:12.027529 28614 tablet_bootstrap.cc:654] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.028370 29334 rpcz_store.cc:276] 0503 14:12:10.408751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.408845 (+ 94us) service_pool.cc:225] Handling call
0503 14:12:12.028152 (+1619307us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:10.453213 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:10.453333 (+ 120us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:11.956435 (+1503102us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:11.956447 (+ 12us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:11.956499 (+ 52us) alter_schema_op.cc:119] START. Timestamp: 1714745531956451
0503 14:12:11.956502 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:11.956723 (+ 221us) log.cc:842] Serialized 173 byte log entry
0503 14:12:12.026787 (+ 70064us) op_driver.cc:444] REPLICATION: finished
0503 14:12:12.027308 (+ 521us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:12.028056 (+ 748us) log.cc:842] Serialized 94 byte log entry
0503 14:12:12.028119 (+ 63us) alter_schema_op.cc:65] Released schema lock
0503 14:12:12.028122 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":455,"prepare.queue_time_us":87,"prepare.run_cpu_time_us":180943,"prepare.run_wall_time_us":1503599,"raft.queue_time_us":211,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":299,"replication_time_us":70254}]]}
W20240503 14:12:12.029021 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b24ab3a5bba04a5092225a981f292222 consensus state has no leader
W20240503 14:12:12.027660 28290 leader_election.cc:341] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: Tablet not RUNNING: STOPPED
W20240503 14:12:12.031631 28290 leader_election.cc:341] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 74df7781cc6a4dd6b6fd78a728ffd71b
I20240503 14:12:12.031831 28290 leader_election.cc:302] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.032433 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet d7c79ffee3954b5c870f4a8aa6379517 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:12.032498 28427 leader_election.cc:341] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.032650 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9ac2833a0bb045dcb2824468d936ffe4
I20240503 14:12:12.032661 29297 raft_consensus.cc:2726] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.033406 29330 leader_election.cc:288] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.037617 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:12.039819 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6dab0296a0894a63afc16ed466c591d5 (table default.Table_de96804c306a4b0e91a2e6bffd5943a8 [id=1b1ed3d3a0b24ced96ba66459ccecc40]) successfully deleted
W20240503 14:12:12.040210 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 44208.5us
I20240503 14:12:12.040705 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1c3e2f0212484ab48f7c25295525231d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.041605 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.042274 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.042524 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7efeff78f7b04eadbc81cf4b5fb025a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.043555 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d28d3c2ea472191ad21355248a2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:12.044355 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7efeff78f7b04eadbc81cf4b5fb025a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.046499 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.046743 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_40fb6c7d8a074d12962a5f9b17b3c14f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.047992 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40fb6c7d8a074d12962a5f9b17b3c14f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:12.049402 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c3e2f0212484ab48f7c25295525231d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.051172 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fc0d4a5f37ae46d9807079682b94ba1d" } modify_external_catalogs: true
I20240503 14:12:12.051393 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fc0d4a5f37ae46d9807079682b94ba1d" } modify_external_catalogs: true
I20240503 14:12:12.055408 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129717ef137c41c5a08d401c64164187" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:12.056752 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d28d3c2ea472191ad21355248a2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.057104 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.057689 29358 tablet.cc:1717] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 4, tablet already version 4
W20240503 14:12:12.057722 28427 leader_election.cc:341] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.058984 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129717ef137c41c5a08d401c64164187" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
W20240503 14:12:12.062247 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7945118bb17b431dac8670007d2d0a18 (table default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7945118bb17b431dac8670007d2d0a18 consensus state has no leader
I20240503 14:12:12.062358 28661 ts_tablet_manager.cc:1888] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.062552 28661 log.cc:1196] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/74df7781cc6a4dd6b6fd78a728ffd71b
I20240503 14:12:12.063081 28661 ts_tablet_manager.cc:1909] T 74df7781cc6a4dd6b6fd78a728ffd71b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:12.063136 29305 tablet.cc:1717] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 4, tablet already version 4
I20240503 14:12:12.063925 28604 tablet_bootstrap.cc:492] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
W20240503 14:12:12.064421 28427 leader_election.cc:341] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:12.065976 28427 leader_election.cc:302] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:12.066179 28604 ts_tablet_manager.cc:1367] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.157s user 0.003s sys 0.011s
I20240503 14:12:12.068379 29334 tablet.cc:1717] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 4, tablet already version 4
I20240503 14:12:12.069463 28603 ts_tablet_manager.cc:1888] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:12.069504 29316 raft_consensus.cc:2726] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.072320 28604 raft_consensus.cc:348] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.065379 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 346) took 2255 ms. Trace:
I20240503 14:12:12.074549 29305 tablet.cc:1717] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 4, tablet already version 4
I20240503 14:12:12.071813 29358 tablet.cc:1717] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 4, tablet already version 4
I20240503 14:12:12.074085 28614 tablet_bootstrap.cc:492] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:12.075017 28614 ts_tablet_manager.cc:1367] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.120s user 0.002s sys 0.011s
I20240503 14:12:12.075503 28603 log.cc:1196] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/da19787034c24ebe929601d49cb21b3b
I20240503 14:12:12.076337 28603 ts_tablet_manager.cc:1909] T da19787034c24ebe929601d49cb21b3b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.077337 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 351) took 2109 ms. Trace:
I20240503 14:12:12.076635 28614 raft_consensus.cc:348] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.078085 28603 rpcz_store.cc:276] 0503 14:12:09.967338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.967844 (+ 506us) service_pool.cc:225] Handling call
0503 14:12:11.958175 (+1990331us) ts_tablet_manager.cc:1120] Deleting tablet da19787034c24ebe929601d49cb21b3b
0503 14:12:11.958186 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.058620 (+100434us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.069282 (+ 10662us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.077292 (+ 8010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1990053}
I20240503 14:12:12.078445 28653 ts_tablet_manager.cc:1888] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.078461 28614 raft_consensus.cc:374] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.078920 28614 raft_consensus.cc:729] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.079633 28603 tablet_replica.cc:330] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.080174 28603 raft_consensus.cc:2229] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.080703 28614 consensus_queue.cc:260] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.082192 28603 raft_consensus.cc:2258] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.084571 28603 ts_tablet_manager.cc:1875] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.085568 29358 tablet.cc:1722] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:12.090521 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0982b99a02ee48f2b6379e7be56c4c02 (table default.Table_79726aaee899484aaf8147ce414cbd87 [id=8cce76d1e8a642a4b051b4c4a854be78]) successfully deleted
I20240503 14:12:12.093906 29330 raft_consensus.cc:482] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.094174 29330 raft_consensus.cc:504] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.094960 29330 leader_election.cc:288] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.095425 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet da19787034c24ebe929601d49cb21b3b (table default.Table_6c996bea4ba44fefbb3bc602c3c76f58 [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:12.080750 28600 tablet_bootstrap.cc:492] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:12.088802 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d28d3c2ea472191ad21355248a2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.089684 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.096388 29358 tablet.cc:1717] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 2, tablet already version 2
I20240503 14:12:12.096936 28600 ts_tablet_manager.cc:1367] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.125s user 0.012s sys 0.004s
I20240503 14:12:12.097206 28604 raft_consensus.cc:374] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.097358 28604 raft_consensus.cc:729] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.098261 28653 log.cc:1196] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/129717ef137c41c5a08d401c64164187
I20240503 14:12:12.099052 28653 ts_tablet_manager.cc:1909] T 129717ef137c41c5a08d401c64164187 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:12.099572 28661 rpcz_store.cc:276] 0503 14:12:09.809744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.809779 (+ 35us) service_pool.cc:225] Handling call
0503 14:12:11.939027 (+2129248us) ts_tablet_manager.cc:1120] Deleting tablet 74df7781cc6a4dd6b6fd78a728ffd71b
0503 14:12:11.939039 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.009887 (+ 70848us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.062206 (+ 52319us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.065318 (+ 3112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129077}
I20240503 14:12:12.099845 28614 ts_tablet_manager.cc:1398] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240503 14:12:12.098435 28600 raft_consensus.cc:348] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.100278 28614 tablet_bootstrap.cc:492] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:12.100234 28600 raft_consensus.cc:374] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.100410 28600 raft_consensus.cc:729] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.101248 28600 consensus_queue.cc:260] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.101981 28614 tablet_bootstrap.cc:654] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.102411 28600 ts_tablet_manager.cc:1398] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:12.103264 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 74df7781cc6a4dd6b6fd78a728ffd71b (table default.Table_e1c42e57b67a4007ad1fa7e9c03eb1c2 [id=b7f4f49135e245dc975ffd7eb0589a8d]) successfully deleted
I20240503 14:12:12.103569 28604 consensus_queue.cc:260] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.103821 28600 tablet_bootstrap.cc:492] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:12.104373 28604 ts_tablet_manager.cc:1398] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.038s user 0.000s sys 0.003s
I20240503 14:12:12.104755 28604 tablet_bootstrap.cc:492] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:12.105386 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 381) took 1793 ms. Trace:
I20240503 14:12:12.105755 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50ad403bd354559b2522a76af1fa722" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.106084 28661 tablet_replica.cc:330] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:12.106300 28661 raft_consensus.cc:2229] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.106496 28661 raft_consensus.cc:2258] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.107115 28661 ts_tablet_manager.cc:1875] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.108362 28653 rpcz_store.cc:276] 0503 14:12:10.312025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.336228 (+ 24203us) service_pool.cc:225] Handling call
0503 14:12:12.021365 (+1685137us) ts_tablet_manager.cc:1120] Deleting tablet 129717ef137c41c5a08d401c64164187
0503 14:12:12.021374 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.044861 (+ 23487us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.078299 (+ 33438us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.105321 (+ 27022us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4340,"tablet-delete.queue_time_us":1679324}
I20240503 14:12:12.109159 28653 tablet_replica.cc:330] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:12.109396 28653 raft_consensus.cc:2229] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.109635 28653 raft_consensus.cc:2258] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.110275 28653 ts_tablet_manager.cc:1875] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:12.111356 28290 leader_election.cc:341] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: f50ad403bd354559b2522a76af1fa722
W20240503 14:12:12.111841 28290 leader_election.cc:341] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.112013 28290 leader_election.cc:302] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.112672 28600 tablet_bootstrap.cc:654] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.113431 29330 raft_consensus.cc:2726] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.129069 28604 tablet_bootstrap.cc:654] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.133587 28661 ts_tablet_manager.cc:1888] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.133915 28661 log.cc:1196] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/f50ad403bd354559b2522a76af1fa722
I20240503 14:12:12.134647 28661 ts_tablet_manager.cc:1909] T f50ad403bd354559b2522a76af1fa722 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:12.141824 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 347) took 2331 ms. Trace:
I20240503 14:12:12.144214 28661 rpcz_store.cc:276] 0503 14:12:09.810062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.810099 (+ 37us) service_pool.cc:225] Handling call
0503 14:12:12.099952 (+2289853us) ts_tablet_manager.cc:1120] Deleting tablet f50ad403bd354559b2522a76af1fa722
0503 14:12:12.099963 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.114778 (+ 14815us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.133389 (+ 18611us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.141758 (+ 8369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2289678}
I20240503 14:12:12.148182 28600 tablet_bootstrap.cc:492] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:12.148569 29281 raft_consensus.cc:482] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.148762 29281 raft_consensus.cc:504] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.149715 29281 leader_election.cc:288] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:12.150420 28600 ts_tablet_manager.cc:1367] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.003s
I20240503 14:12:12.150714 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962df44408484e3f8d7b7138f27c1a66" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:12.151378 28150 leader_election.cc:341] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.152019 28600 raft_consensus.cc:348] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.152287 28653 ts_tablet_manager.cc:1888] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.153822 28600 raft_consensus.cc:374] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.154067 28600 raft_consensus.cc:729] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.150714 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962df44408484e3f8d7b7138f27c1a66" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.154489 28653 log.cc:1196] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6371e68ee85a47cdb0e8e21f7124d94a
I20240503 14:12:12.155066 28600 consensus_queue.cc:260] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.155929 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 350) took 1966 ms. Trace:
I20240503 14:12:12.157425 28604 tablet_bootstrap.cc:492] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:12.157744 28604 ts_tablet_manager.cc:1367] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.006s
I20240503 14:12:12.157914 28600 ts_tablet_manager.cc:1398] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:12.158399 28600 tablet_bootstrap.cc:492] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:12.159029 28604 raft_consensus.cc:348] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.156751 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 129717ef137c41c5a08d401c64164187 (table default.Table_36c01c2a0d944899bf662092e1cc816d [id=f9add48fee4b4ca89754d81057f832e4]) successfully deleted
I20240503 14:12:12.160003 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet f50ad403bd354559b2522a76af1fa722 (table default.Table_5eec9210201643c3a011af892345d421 [id=f4c76f2fa96d41c491fb0ac30f49e43f]) successfully deleted
I20240503 14:12:12.162824 28603 ts_tablet_manager.cc:1888] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.163170 28603 log.cc:1196] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ffbd3291c56b4c13813c80d6775afe6f
I20240503 14:12:12.163681 28603 ts_tablet_manager.cc:1909] T ffbd3291c56b4c13813c80d6775afe6f P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.165023 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 352) took 2194 ms. Trace:
I20240503 14:12:12.166935 28603 rpcz_store.cc:276] 0503 14:12:09.970092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.970154 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:12.079546 (+2109392us) ts_tablet_manager.cc:1120] Deleting tablet ffbd3291c56b4c13813c80d6775afe6f
0503 14:12:12.079557 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.126148 (+ 46591us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.162670 (+ 36522us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.164950 (+ 2280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2109120}
I20240503 14:12:12.167827 28603 tablet_replica.cc:330] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.168244 28603 raft_consensus.cc:2229] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.159317 28604 raft_consensus.cc:374] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:12:12.169135 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1ebec4622ef4950bd1c70451958c59b consensus state has no leader
I20240503 14:12:12.169327 28604 raft_consensus.cc:729] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.169432 28603 raft_consensus.cc:2258] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.169858 28604 consensus_queue.cc:260] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.170177 28603 ts_tablet_manager.cc:1875] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.170622 28604 ts_tablet_manager.cc:1398] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:12:12.171066 28604 tablet_bootstrap.cc:492] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:12.171717 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ab664e165be64811ae894db7bab917d9" } new_table_name: "default.Table_4e0a1dcdeddb46cfac04de76adfc27e5" modify_external_catalogs: true
I20240503 14:12:12.172324 28661 rpcz_store.cc:276] 0503 14:12:10.189794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.189863 (+ 69us) service_pool.cc:225] Handling call
0503 14:12:12.155692 (+1965829us) ts_tablet_manager.cc:1120] Deleting tablet e0175b38c1e0464a9a6208cfaf68ca0d
0503 14:12:12.155706 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.155863 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1965249}
I20240503 14:12:12.175879 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 351) took 1984 ms. Trace:
I20240503 14:12:12.176422 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_05db2dcd653343c0ac8992415e36c7d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.177860 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05db2dcd653343c0ac8992415e36c7d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.179433 28661 rpcz_store.cc:276] 0503 14:12:10.191522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.191594 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:12.172643 (+1981049us) ts_tablet_manager.cc:1120] Deleting tablet 1603e2360041427f8635782e812b6431
0503 14:12:12.172653 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.175804 (+ 3151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1979584}
I20240503 14:12:12.181351 28604 tablet_bootstrap.cc:654] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.181689 28033 catalog_manager.cc:5484] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:12.176039 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_42ee02ce6afe4ddb8480ec52a7d8993f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.181667 28661 tablet_replica.cc:330] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:12.186316 28661 raft_consensus.cc:2229] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.186700 28661 raft_consensus.cc:2258] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.187069 28614 tablet_bootstrap.cc:492] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:12.187291 28614 ts_tablet_manager.cc:1367] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.087s user 0.009s sys 0.014s
I20240503 14:12:12.187635 28661 ts_tablet_manager.cc:1875] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.188429 28614 raft_consensus.cc:348] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.188702 28614 raft_consensus.cc:374] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.188882 28614 raft_consensus.cc:729] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.189217 28614 consensus_queue.cc:260] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.190972 28614 ts_tablet_manager.cc:1398] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20240503 14:12:12.191783 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42ee02ce6afe4ddb8480ec52a7d8993f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.192381 28614 tablet_bootstrap.cc:492] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:12.193253 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:12.195286 29330 raft_consensus.cc:482] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.195524 29330 raft_consensus.cc:504] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.196282 29330 leader_election.cc:288] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.197350 28614 tablet_bootstrap.cc:654] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.197279 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129717ef137c41c5a08d401c64164187" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.197625 28489 raft_consensus.cc:2379] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:12.199121 28603 ts_tablet_manager.cc:1888] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.201385 28604 tablet_bootstrap.cc:492] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:12.201632 28604 ts_tablet_manager.cc:1367] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20240503 14:12:12.201658 28603 log.cc:1196] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/40da5f5fff94462d8af740f3ee337e76
I20240503 14:12:12.202309 28603 ts_tablet_manager.cc:1909] T 40da5f5fff94462d8af740f3ee337e76 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.202711 28604 raft_consensus.cc:348] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.203027 28604 raft_consensus.cc:374] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.203142 28604 raft_consensus.cc:729] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.203454 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 354) took 2229 ms. Trace:
I20240503 14:12:12.203516 28604 consensus_queue.cc:260] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.203660 28603 rpcz_store.cc:276] 0503 14:12:09.973903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.973959 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:12.167691 (+2193732us) ts_tablet_manager.cc:1120] Deleting tablet 40da5f5fff94462d8af740f3ee337e76
0503 14:12:12.167705 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.186901 (+ 19196us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.198934 (+ 12033us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.203408 (+ 4474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2193341}
I20240503 14:12:12.204913 28604 ts_tablet_manager.cc:1398] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:12.208428 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7ca19dd15f48a68ace7fefc7ae660f (DEFAULT_TABLE table=default.Table_f7bbe9e54e404d698feb87bac95370d0 [id=84f355c001064d28a408a8012a24df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.208884 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 99aa7e59c45c45c7bc1f726c97426b89 (DEFAULT_TABLE table=default.Table_1d2243e88c784b929de2fc4d94804219 [id=8894157a06784f4288235959f27849b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.209040 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7ca19dd15f48a68ace7fefc7ae660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.209355 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99aa7e59c45c45c7bc1f726c97426b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.209420 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7ca19dd15f48a68ace7fefc7ae660f (DEFAULT_TABLE table=default.Table_f7bbe9e54e404d698feb87bac95370d0 [id=84f355c001064d28a408a8012a24df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.209862 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7ca19dd15f48a68ace7fefc7ae660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.212275 28461 tablet_service.cc:1467] Processing CreateTablet for tablet e69e01bdbae349d0a61a295b6f6352f0 (DEFAULT_TABLE table=default.Table_b5f59b1cb2f440d4ae381befe0809756 [id=3cf0346f342943c89b600c0e8f1428e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.212463 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 34fa067359a640eeba8c80d08371904e (DEFAULT_TABLE table=default.Table_021c165af8934008b2bb8bb0487c0af4 [id=dc09b38a81a242a990d0f87cd533e306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.213109 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 34fa067359a640eeba8c80d08371904e (DEFAULT_TABLE table=default.Table_021c165af8934008b2bb8bb0487c0af4 [id=dc09b38a81a242a990d0f87cd533e306]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:12.213129 28150 leader_election.cc:341] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.213375 28150 leader_election.cc:302] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.213575 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fa067359a640eeba8c80d08371904e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.209420 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 99aa7e59c45c45c7bc1f726c97426b89 (DEFAULT_TABLE table=default.Table_1d2243e88c784b929de2fc4d94804219 [id=8894157a06784f4288235959f27849b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.214527 28325 tablet_service.cc:1467] Processing CreateTablet for tablet e330b28d642a47e69e427b56a90d9807 (DEFAULT_TABLE table=default.Table_cca08ca8fa2f4f87a3793867e49f7b6b [id=67d5b61344fa4e5d8fa86ba130193877]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.214901 28333 tablet_service.cc:1467] Processing CreateTablet for tablet e69e01bdbae349d0a61a295b6f6352f0 (DEFAULT_TABLE table=default.Table_b5f59b1cb2f440d4ae381befe0809756 [id=3cf0346f342943c89b600c0e8f1428e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.215221 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e330b28d642a47e69e427b56a90d9807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.216161 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 007c84fd9d9247728b0babd48689aba8 (DEFAULT_TABLE table=default.Table_446afb3f188b43f1aea50e0337a2c793 [id=f632a2b3ec204ee5b2baad153068b76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.216977 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007c84fd9d9247728b0babd48689aba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.217034 28460 tablet_service.cc:1467] Processing CreateTablet for tablet aa3ce05b422a4166bfac468acb84e4cc (DEFAULT_TABLE table=default.Table_ab664e165be64811ae894db7bab917d9 [id=c53d59b2f4634fd7b3c26f749d9acc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.217867 28327 tablet_service.cc:1467] Processing CreateTablet for tablet aa3ce05b422a4166bfac468acb84e4cc (DEFAULT_TABLE table=default.Table_ab664e165be64811ae894db7bab917d9 [id=c53d59b2f4634fd7b3c26f749d9acc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.217995 28603 tablet_replica.cc:330] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.218158 28455 tablet_service.cc:1467] Processing CreateTablet for tablet e330b28d642a47e69e427b56a90d9807 (DEFAULT_TABLE table=default.Table_cca08ca8fa2f4f87a3793867e49f7b6b [id=67d5b61344fa4e5d8fa86ba130193877]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.218291 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3ce05b422a4166bfac468acb84e4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.218616 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 725b4c9a47c0458ba188c0cf62f3fe48 (DEFAULT_TABLE table=default.Table_3ede1f6683664f5d9956c6f825d3d7c8 [id=ec20cb1644174d6c8a21107acbf0a040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.218976 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 007c84fd9d9247728b0babd48689aba8 (DEFAULT_TABLE table=default.Table_446afb3f188b43f1aea50e0337a2c793 [id=f632a2b3ec204ee5b2baad153068b76c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:12.220983 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e0175b38c1e0464a9a6208cfaf68ca0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0175b38c1e0464a9a6208cfaf68ca0d already in progress: creating tablet
I20240503 14:12:12.221437 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69e01bdbae349d0a61a295b6f6352f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.221837 28337 tablet_service.cc:1467] Processing CreateTablet for tablet d4c8a97076a44f9eaa3a15026f90a304 (DEFAULT_TABLE table=default.Table_80d78826082349c882ea2c0ca66e0141 [id=bc70f87f0df24f569c3cc1c65a86f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.222291 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c8a97076a44f9eaa3a15026f90a304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.222812 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69e01bdbae349d0a61a295b6f6352f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.221474 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 725b4c9a47c0458ba188c0cf62f3fe48 (DEFAULT_TABLE table=default.Table_3ede1f6683664f5d9956c6f825d3d7c8 [id=ec20cb1644174d6c8a21107acbf0a040]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:12.224283 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1603e2360041427f8635782e812b6431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1603e2360041427f8635782e812b6431 already in progress: creating tablet
I20240503 14:12:12.224399 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b4c9a47c0458ba188c0cf62f3fe48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.224560 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ffbd3291c56b4c13813c80d6775afe6f (table default.Table_f99793320ab94248abb70c15ee57ca02 [id=cde896fd6d7345f998085c10bbdee418]) successfully deleted
I20240503 14:12:12.224915 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 40da5f5fff94462d8af740f3ee337e76 (table default.Table_63619685889a4263ae065b254160cf30 [id=6a81b734738e4d998215938d5ca5dad7]) successfully deleted
I20240503 14:12:12.227290 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007c84fd9d9247728b0babd48689aba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.227447 28463 tablet_service.cc:1467] Processing CreateTablet for tablet d4c8a97076a44f9eaa3a15026f90a304 (DEFAULT_TABLE table=default.Table_80d78826082349c882ea2c0ca66e0141 [id=bc70f87f0df24f569c3cc1c65a86f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.228029 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c8a97076a44f9eaa3a15026f90a304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.218307 28603 raft_consensus.cc:2229] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:12.232815 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e330b28d642a47e69e427b56a90d9807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.236490 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3ce05b422a4166bfac468acb84e4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.236554 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8e709ec40c1b4306944f2be34700e617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.237277 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e709ec40c1b4306944f2be34700e617 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.239838 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b4c9a47c0458ba188c0cf62f3fe48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.247344 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fa067359a640eeba8c80d08371904e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.248991 28603 pending_rounds.cc:62] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:12.252205 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99aa7e59c45c45c7bc1f726c97426b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.254799 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 54f08aac9a7147ebaffda4262a57a86a
I20240503 14:12:12.255797 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a6709087fc714b21979fd6349c864785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.256165 28603 pending_rounds.cc:68] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597696734105600 op_type: NO_OP noop_request { }
I20240503 14:12:12.256378 28603 raft_consensus.cc:2866] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240503 14:12:12.256505 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6709087fc714b21979fd6349c864785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.256935 28603 raft_consensus.cc:2258] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.258031 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d7c79ffee3954b5c870f4a8aa6379517
I20240503 14:12:12.258666 28027 catalog_manager.cc:6554] ReplaceTablet: tablet d7c79ffee3954b5c870f4a8aa6379517 deleted and replaced by tablet c9c0540960ac45aabeaf13f888caf238
I20240503 14:12:12.259187 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a888de5f6a40467db886c3141d2a87cc" } modify_external_catalogs: true
I20240503 14:12:12.259443 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a888de5f6a40467db886c3141d2a87cc" } modify_external_catalogs: true
I20240503 14:12:12.260370 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:12.260833 28031 catalog_manager.cc:6554] ReplaceTablet: tablet 0af0342e0e0b47499a9322b5b2cb3d03 deleted and replaced by tablet 70f0e07d32084999a3ff42f69bb1ca2f
I20240503 14:12:12.261303 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_354b5d04e8994484bf17eda27e660238" } new_table_name: "default.Table_2d7ecf8640f64d82908807b4c4d4fd7f" modify_external_catalogs: true
I20240503 14:12:12.263049 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 605531ed29214b1ca27011c5f5e96b11
I20240503 14:12:12.263805 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet 007a892ef8504b848a400c05740eaa0c from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:12.264560 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa3ce05b422a4166bfac468acb84e4cc consensus state has no leader
I20240503 14:12:12.264890 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet e0175b38c1e0464a9a6208cfaf68ca0d from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:12.265699 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1c3e2f0212484ab48f7c25295525231d" } new_table_name: "default.Table_9ecfb667ad96405c894e48eda793fada" modify_external_catalogs: true
I20240503 14:12:12.268527 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2e27287910d04e1b9c08ae3eca4739b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.269297 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e27287910d04e1b9c08ae3eca4739b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.274363 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:12.274930 28025 catalog_manager.cc:5484] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:12.275475 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]). Expected version 2 got 0
I20240503 14:12:12.275852 28025 catalog_manager.cc:5484] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:12.281766 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]). Expected version 1 got 0
I20240503 14:12:12.283676 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_21ea716e646a43279621abff726dedd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.285033 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21ea716e646a43279621abff726dedd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.293617 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f08aac9a7147ebaffda4262a57a86a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:12.298277 28290 leader_election.cc:341] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 129717ef137c41c5a08d401c64164187
I20240503 14:12:12.298491 28290 leader_election.cc:302] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.298766 28603 ts_tablet_manager.cc:1875] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.298763 29281 raft_consensus.cc:2726] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.299510 29330 raft_consensus.cc:3037] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:12.300668 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7ca19dd15f48a68ace7fefc7ae660f (DEFAULT_TABLE table=default.Table_f7bbe9e54e404d698feb87bac95370d0 [id=84f355c001064d28a408a8012a24df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.301155 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7ca19dd15f48a68ace7fefc7ae660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.302922 29330 raft_consensus.cc:2726] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.304687 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 with delete_type TABLET_DATA_DELETED (replaced by tablet 70f0e07d32084999a3ff42f69bb1ca2f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.305130 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet f50ad403bd354559b2522a76af1fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.306324 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 34fa067359a640eeba8c80d08371904e (DEFAULT_TABLE table=default.Table_021c165af8934008b2bb8bb0487c0af4 [id=dc09b38a81a242a990d0f87cd533e306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.306880 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fa067359a640eeba8c80d08371904e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.312572 28193 tablet_service.cc:1467] Processing CreateTablet for tablet e330b28d642a47e69e427b56a90d9807 (DEFAULT_TABLE table=default.Table_cca08ca8fa2f4f87a3793867e49f7b6b [id=67d5b61344fa4e5d8fa86ba130193877]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.313148 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e330b28d642a47e69e427b56a90d9807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.314024 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c0540960ac45aabeaf13f888caf238) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.315141 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_96e46f964a63484bb1f66f90e65c72db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.315889 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96e46f964a63484bb1f66f90e65c72db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.316012 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 007c84fd9d9247728b0babd48689aba8 (DEFAULT_TABLE table=default.Table_446afb3f188b43f1aea50e0337a2c793 [id=f632a2b3ec204ee5b2baad153068b76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.316591 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007c84fd9d9247728b0babd48689aba8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:12.317416 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e680f189add4b5c96de4046321aad2d consensus state has no leader
I20240503 14:12:12.317867 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fc57203104f749f6be76b0d76e46e8ee" } new_table_name: "default.Table_c895a23bc5d64550b963a2b67358a358" modify_external_catalogs: true
W20240503 14:12:12.319429 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 has no consensus state
I20240503 14:12:12.319890 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6673b8d0a18d42c3a18f6e611c596580" } modify_external_catalogs: true
I20240503 14:12:12.320116 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6673b8d0a18d42c3a18f6e611c596580" } modify_external_catalogs: true
I20240503 14:12:12.320246 28182 tablet_service.cc:1467] Processing CreateTablet for tablet e69e01bdbae349d0a61a295b6f6352f0 (DEFAULT_TABLE table=default.Table_b5f59b1cb2f440d4ae381befe0809756 [id=3cf0346f342943c89b600c0e8f1428e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.320792 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69e01bdbae349d0a61a295b6f6352f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.321679 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba
I20240503 14:12:12.322055 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 605531ed29214b1ca27011c5f5e96b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.322536 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e88623765c84939b082bcf13612fd61" } modify_external_catalogs: true
I20240503 14:12:12.322744 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e88623765c84939b082bcf13612fd61" } modify_external_catalogs: true
I20240503 14:12:12.323603 28180 tablet_service.cc:1467] Processing CreateTablet for tablet aa3ce05b422a4166bfac468acb84e4cc (DEFAULT_TABLE table=default.Table_ab664e165be64811ae894db7bab917d9 [id=c53d59b2f4634fd7b3c26f749d9acc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.324170 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3ce05b422a4166bfac468acb84e4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.327139 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 725b4c9a47c0458ba188c0cf62f3fe48 (DEFAULT_TABLE table=default.Table_3ede1f6683664f5d9956c6f825d3d7c8 [id=ec20cb1644174d6c8a21107acbf0a040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.327381 28603 ts_tablet_manager.cc:1888] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:12.327558 28603 log.cc:1196] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6dab0296a0894a63afc16ed466c591d5
I20240503 14:12:12.327770 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b4c9a47c0458ba188c0cf62f3fe48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.328276 28603 ts_tablet_manager.cc:1909] T 6dab0296a0894a63afc16ed466c591d5 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.328682 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 353) took 2355 ms. Trace:
I20240503 14:12:12.328809 28603 rpcz_store.cc:276] 0503 14:12:09.973691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.974386 (+ 695us) service_pool.cc:225] Handling call
0503 14:12:12.217898 (+2243512us) ts_tablet_manager.cc:1120] Deleting tablet 6dab0296a0894a63afc16ed466c591d5
0503 14:12:12.217908 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.315679 (+ 97771us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.327187 (+ 11508us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.328626 (+ 1439us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2243247}
I20240503 14:12:12.329808 29317 raft_consensus.cc:482] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.329999 29317 raft_consensus.cc:504] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.330808 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.330873 29317 leader_election.cc:288] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.331321 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.331627 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f08aac9a7147ebaffda4262a57a86a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.331719 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605531ed29214b1ca27011c5f5e96b11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.331983 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c0540960ac45aabeaf13f888caf238) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.332304 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 with delete_type TABLET_DATA_DELETED (replaced by tablet 70f0e07d32084999a3ff42f69bb1ca2f) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:12.332337 28427 leader_election.cc:341] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.332577 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 605531ed29214b1ca27011c5f5e96b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.333473 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 99aa7e59c45c45c7bc1f726c97426b89 (DEFAULT_TABLE table=default.Table_1d2243e88c784b929de2fc4d94804219 [id=8894157a06784f4288235959f27849b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.334029 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99aa7e59c45c45c7bc1f726c97426b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.336264 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605531ed29214b1ca27011c5f5e96b11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:12.336621 28215 raft_consensus.cc:2454] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:12.331275 28195 tablet_service.cc:1467] Processing CreateTablet for tablet d4c8a97076a44f9eaa3a15026f90a304 (DEFAULT_TABLE table=default.Table_80d78826082349c882ea2c0ca66e0141 [id=bc70f87f0df24f569c3cc1c65a86f667]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.338270 29317 raft_consensus.cc:482] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:12.338444 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 52567.8us
I20240503 14:12:12.338498 29317 raft_consensus.cc:504] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.339294 29317 leader_election.cc:288] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:12.340317 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c8a97076a44f9eaa3a15026f90a304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.340822 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6dab0296a0894a63afc16ed466c591d5 (table default.Table_de96804c306a4b0e91a2e6bffd5943a8 [id=1b1ed3d3a0b24ced96ba66459ccecc40]) successfully deleted
I20240503 14:12:12.340925 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:12.341631 28427 leader_election.cc:341] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.342389 29281 raft_consensus.cc:482] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.349758 28427 leader_election.cc:302] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:12.350173 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet c816f3b05cfb4496984f6cb28d310cc4 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:12.350543 29317 raft_consensus.cc:2781] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:12.350673 29317 raft_consensus.cc:482] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:12.350800 29317 raft_consensus.cc:3037] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:12.354099 29317 raft_consensus.cc:504] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.356652 29317 leader_election.cc:288] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.357648 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605531ed29214b1ca27011c5f5e96b11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:12.358351 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 350) took 2391 ms. Trace:
W20240503 14:12:12.358356 28427 leader_election.cc:341] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.358642 28603 rpcz_store.cc:276] 0503 14:12:09.967224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.979896 (+ 12672us) service_pool.cc:225] Handling call
0503 14:12:12.358127 (+2378231us) ts_tablet_manager.cc:1120] Deleting tablet 1603e2360041427f8635782e812b6431
0503 14:12:12.358140 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.358282 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2377632}
I20240503 14:12:12.359252 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f03bf91cf5414208b6767482223bc240 (table default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc]). Expected version 1 got 0
I20240503 14:12:12.359378 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 356) took 2375 ms. Trace:
I20240503 14:12:12.359436 29281 raft_consensus.cc:504] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.359530 28603 rpcz_store.cc:276] 0503 14:12:09.983805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.983859 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:12.359213 (+2375354us) ts_tablet_manager.cc:1120] Deleting tablet a884cf4ea11e48ef941722b79ad50075
0503 14:12:12.359224 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.359314 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375121}
I20240503 14:12:12.359805 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]). Expected version 1 got 0
I20240503 14:12:12.360144 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 355) took 2377 ms. Trace:
I20240503 14:12:12.360301 28603 rpcz_store.cc:276] 0503 14:12:09.983111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:09.983174 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:12.359896 (+2376722us) ts_tablet_manager.cc:1120] Deleting tablet 95d9985eecda4f3194c3251216aa58ee
0503 14:12:12.359905 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.360056 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375515}
I20240503 14:12:12.360749 28032 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_4c09a9ca5e3d4e01820ece646f161b1c [id=874fc82c70674371a8be82614bb6fcfb] alter complete (version 4)
I20240503 14:12:12.360816 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 366) took 2275 ms. Trace:
I20240503 14:12:12.360983 29281 leader_election.cc:288] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:12.360950 28603 rpcz_store.cc:276] 0503 14:12:10.085094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.085172 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:12.360644 (+2275472us) ts_tablet_manager.cc:1120] Deleting tablet 02d45788ae644a8581acca54cc55dd3f
0503 14:12:12.360653 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.360764 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2275135}
I20240503 14:12:12.361166 29281 raft_consensus.cc:482] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.361500 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 367) took 2274 ms. Trace:
I20240503 14:12:12.361634 28603 rpcz_store.cc:276] 0503 14:12:10.087044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.087112 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:12.361336 (+2274224us) ts_tablet_manager.cc:1120] Deleting tablet 3d54cc5c03034a6685c1da34f9bd65ac
0503 14:12:12.361346 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.361439 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272660}
I20240503 14:12:12.361757 29281 raft_consensus.cc:504] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.362115 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 368) took 2265 ms. Trace:
I20240503 14:12:12.362247 28603 rpcz_store.cc:276] 0503 14:12:10.096677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.108186 (+ 11509us) service_pool.cc:225] Handling call
0503 14:12:12.361957 (+2253771us) ts_tablet_manager.cc:1120] Deleting tablet 0bd5234791524eddb7edd493e39fb993
0503 14:12:12.361965 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.362058 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2253291}
I20240503 14:12:12.362571 29281 leader_election.cc:288] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:12.363210 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:12.363472 28215 raft_consensus.cc:2454] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:12.364670 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff1958dc488e4854a8c8e74da9fab243 consensus state has no leader
I20240503 14:12:12.366385 28427 leader_election.cc:302] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:12.366818 28603 tablet_replica.cc:330] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.367070 29317 raft_consensus.cc:2781] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:12.367101 28603 raft_consensus.cc:2229] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.367209 29317 raft_consensus.cc:482] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:12.367297 29317 raft_consensus.cc:3037] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:12.367326 28603 raft_consensus.cc:2258] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.368147 28603 ts_tablet_manager.cc:1875] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.368460 28600 tablet_bootstrap.cc:654] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.369314 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 962df44408484e3f8d7b7138f27c1a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e4a544fc27624dcc8a4cb003e59408f2) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.370637 29317 raft_consensus.cc:504] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.372334 29317 leader_election.cc:288] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:12.372452 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:12.373068 28427 leader_election.cc:341] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:12.375280 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1603e2360041427f8635782e812b6431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1603e2360041427f8635782e812b6431 already in progress: creating tablet
I20240503 14:12:12.375838 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605531ed29214b1ca27011c5f5e96b11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:12.377746 28215 raft_consensus.cc:3037] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:12.376816 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_21dbc608308a4881af80133c4e9e04fd" } new_table_name: "default.Table_29f03216d162412bb8421c8c6fe741e4" modify_external_catalogs: true
I20240503 14:12:12.378252 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet daf5d153ab5142c0902cc6e96629f412
I20240503 14:12:12.381336 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc7d96357743089372baf71654b1e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b90e9563b4c4cef8ee7964eaa96e285) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.381978 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:12.378643 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d933cabbaf9146438f67771682480e59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.382934 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d933cabbaf9146438f67771682480e59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.382709 28212 raft_consensus.cc:3037] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:12.379168 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0a7f788e080745b2bf2fe782dfae1bbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.384789 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a7f788e080745b2bf2fe782dfae1bbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.379491 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_71db0c0f4c9f475fb810a40fff00d5a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.386279 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71db0c0f4c9f475fb810a40fff00d5a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.388172 28603 ts_tablet_manager.cc:1888] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:12.388367 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a884cf4ea11e48ef941722b79ad50075 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a884cf4ea11e48ef941722b79ad50075 already in progress: creating tablet
I20240503 14:12:12.388451 28603 log.cc:1196] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/75dafd98b05d4c63bbf30165a2ac8d1e
I20240503 14:12:12.388885 28603 ts_tablet_manager.cc:1909] T 75dafd98b05d4c63bbf30165a2ac8d1e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:12.388921 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 95d9985eecda4f3194c3251216aa58ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95d9985eecda4f3194c3251216aa58ee already in progress: creating tablet
I20240503 14:12:12.389890 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 375) took 2255 ms. Trace:
I20240503 14:12:12.390091 28212 raft_consensus.cc:2454] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:12.390420 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 02d45788ae644a8581acca54cc55dd3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d45788ae644a8581acca54cc55dd3f already in progress: creating tablet
I20240503 14:12:12.390305 28603 rpcz_store.cc:276] 0503 14:12:10.134499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.135428 (+ 929us) service_pool.cc:225] Handling call
0503 14:12:12.366713 (+2231285us) ts_tablet_manager.cc:1120] Deleting tablet 75dafd98b05d4c63bbf30165a2ac8d1e
0503 14:12:12.366728 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.371165 (+ 4437us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.387989 (+ 16824us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.389820 (+ 1831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230420}
I20240503 14:12:12.390913 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 382) took 2108 ms. Trace:
I20240503 14:12:12.391888 28603 rpcz_store.cc:276] 0503 14:12:10.282084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.282639 (+ 555us) service_pool.cc:225] Handling call
0503 14:12:12.390642 (+2108003us) ts_tablet_manager.cc:1120] Deleting tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
0503 14:12:12.390654 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.390814 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068834}
I20240503 14:12:12.392454 28427 leader_election.cc:302] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:12.393039 29317 raft_consensus.cc:2781] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:12.393249 29317 raft_consensus.cc:686] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
W20240503 14:12:12.393743 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 3d54cc5c03034a6685c1da34f9bd65ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d54cc5c03034a6685c1da34f9bd65ac already in progress: creating tablet
I20240503 14:12:12.393630 29317 consensus_queue.cc:237] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:12.394013 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0bd5234791524eddb7edd493e39fb993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bd5234791524eddb7edd493e39fb993 already in progress: creating tablet
I20240503 14:12:12.394243 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 75dafd98b05d4c63bbf30165a2ac8d1e (table default.Table_986d778bcd814c9c87dbd18d1f485cc9 [id=5963eb580afa4dc18cf4e472ed1fb76c]) successfully deleted
W20240503 14:12:12.394476 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a already in progress: creating tablet
I20240503 14:12:12.395391 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 384) took 2056 ms. Trace:
I20240503 14:12:12.395584 28603 rpcz_store.cc:276] 0503 14:12:10.339000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.339162 (+ 162us) service_pool.cc:225] Handling call
0503 14:12:12.392808 (+2053646us) ts_tablet_manager.cc:1120] Deleting tablet a035a5249f8348b591b27ba3fa7ae43a
0503 14:12:12.392823 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.395300 (+ 2477us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045056}
I20240503 14:12:12.396535 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 383) took 2065 ms. Trace:
I20240503 14:12:12.396737 28603 rpcz_store.cc:276] 0503 14:12:10.331023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.332407 (+ 1384us) service_pool.cc:225] Handling call
0503 14:12:12.396295 (+2063888us) ts_tablet_manager.cc:1120] Deleting tablet 8c6202333af943cbbc0d09553c1c193c
0503 14:12:12.396313 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.396441 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942079}
I20240503 14:12:12.398039 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 388) took 1941 ms. Trace:
I20240503 14:12:12.398254 28603 rpcz_store.cc:276] 0503 14:12:10.456735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.458024 (+ 1289us) service_pool.cc:225] Handling call
0503 14:12:12.397257 (+1939233us) ts_tablet_manager.cc:1120] Deleting tablet 20b176a071ba4f3fbff2a64d6e46027a
0503 14:12:12.397828 (+ 571us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.397947 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1913718}
I20240503 14:12:12.398854 28603 tablet_replica.cc:330] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.407223 28215 raft_consensus.cc:2454] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:12.408026 28427 leader_election.cc:302] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:12.408807 29364 raft_consensus.cc:2781] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:12.408996 29364 raft_consensus.cc:686] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:12.409404 29364 consensus_queue.cc:237] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.410360 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet daf5d153ab5142c0902cc6e96629f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.410566 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet daf5d153ab5142c0902cc6e96629f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.410989 28603 raft_consensus.cc:2229] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:12.411499 28603 raft_consensus.cc:2258] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.412256 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3d137ef4cfc2476cbc70f1cc72ad89e6
I20240503 14:12:12.412464 28603 ts_tablet_manager.cc:1875] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.419140 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet daf5d153ab5142c0902cc6e96629f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:12.419335 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a035a5249f8348b591b27ba3fa7ae43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a035a5249f8348b591b27ba3fa7ae43a already in progress: creating tablet
W20240503 14:12:12.419746 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 8c6202333af943cbbc0d09553c1c193c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6202333af943cbbc0d09553c1c193c already in progress: creating tablet
W20240503 14:12:12.420034 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 20b176a071ba4f3fbff2a64d6e46027a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20b176a071ba4f3fbff2a64d6e46027a already in progress: creating tablet
I20240503 14:12:12.420910 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.421334 28603 ts_tablet_manager.cc:1888] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:12.421469 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.421579 28603 log.cc:1196] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/13c875ec8c5e49d1bb20a23dd7a21902
I20240503 14:12:12.422333 28603 ts_tablet_manager.cc:1909] T 13c875ec8c5e49d1bb20a23dd7a21902 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.423310 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 401) took 1916 ms. Trace:
I20240503 14:12:12.432248 28603 rpcz_store.cc:276] 0503 14:12:10.506987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.507192 (+ 205us) service_pool.cc:225] Handling call
0503 14:12:12.398732 (+1891540us) ts_tablet_manager.cc:1120] Deleting tablet 13c875ec8c5e49d1bb20a23dd7a21902
0503 14:12:12.398741 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.416755 (+ 18014us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.421144 (+ 4389us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.423268 (+ 2124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891279}
I20240503 14:12:12.433216 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 400) took 1926 ms. Trace:
I20240503 14:12:12.433485 28603 rpcz_store.cc:276] 0503 14:12:10.506871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.508056 (+ 1185us) service_pool.cc:225] Handling call
0503 14:12:12.432954 (+1924898us) ts_tablet_manager.cc:1120] Deleting tablet 00b52ea336524beea2f255c8720f7472
0503 14:12:12.432968 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.433148 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1924652}
I20240503 14:12:12.433681 28025 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_61bbab05d19346f4b659073493a0a3c6 [id=b1f8924b4e464e11aec7070b0e7c3951] alter complete (version 2)
I20240503 14:12:12.434273 28603 tablet_replica.cc:330] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.434623 28603 raft_consensus.cc:2229] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.435223 28603 raft_consensus.cc:2258] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.435360 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet c816f3b05cfb4496984f6cb28d310cc4
I20240503 14:12:12.435868 28033 catalog_manager.cc:6554] ReplaceTablet: tablet c816f3b05cfb4496984f6cb28d310cc4 deleted and replaced by tablet 8097ba5606cf42d68448833de36d85c7
I20240503 14:12:12.435918 28653 ts_tablet_manager.cc:1909] T 6371e68ee85a47cdb0e8e21f7124d94a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:12.436497 28603 ts_tablet_manager.cc:1875] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.437971 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 385) took 2074 ms. Trace:
I20240503 14:12:12.438153 28653 rpcz_store.cc:276] 0503 14:12:10.363018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.363095 (+ 77us) service_pool.cc:225] Handling call
0503 14:12:12.109029 (+1745934us) ts_tablet_manager.cc:1120] Deleting tablet 6371e68ee85a47cdb0e8e21f7124d94a
0503 14:12:12.109041 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.127973 (+ 18932us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.152074 (+ 24101us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.437926 (+285852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745698}
I20240503 14:12:12.438870 28653 tablet_replica.cc:330] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:12.439316 28653 raft_consensus.cc:2229] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:12.439790 28653 raft_consensus.cc:2258] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.443425 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 with delete_type TABLET_DATA_DELETED (replaced by tablet 70f0e07d32084999a3ff42f69bb1ca2f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.444447 28603 ts_tablet_manager.cc:1888] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:12.444661 28603 log.cc:1196] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e9609112657f427cbeeb3ef8f28cddf0
I20240503 14:12:12.445230 28603 ts_tablet_manager.cc:1909] T e9609112657f427cbeeb3ef8f28cddf0 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.445355 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.445858 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.446164 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 399) took 1939 ms. Trace:
I20240503 14:12:12.446275 28603 rpcz_store.cc:276] 0503 14:12:10.506726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.514638 (+ 7912us) service_pool.cc:225] Handling call
0503 14:12:12.434164 (+1919526us) ts_tablet_manager.cc:1120] Deleting tablet e9609112657f427cbeeb3ef8f28cddf0
0503 14:12:12.434177 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.440283 (+ 6106us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.444304 (+ 4021us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.446085 (+ 1781us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919076}
I20240503 14:12:12.446898 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 387) took 1990 ms. Trace:
I20240503 14:12:12.447016 28603 rpcz_store.cc:276] 0503 14:12:10.456606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.458217 (+ 1611us) service_pool.cc:225] Handling call
0503 14:12:12.446614 (+1988397us) ts_tablet_manager.cc:1120] Deleting tablet 0982b99a02ee48f2b6379e7be56c4c02
0503 14:12:12.446625 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.446819 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1931197}
I20240503 14:12:12.448464 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a28d952dbd9141a9a113c42bcbc27a14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.449122 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a28d952dbd9141a9a113c42bcbc27a14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.449765 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c0540960ac45aabeaf13f888caf238) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.454030 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8097ba5606cf42d68448833de36d85c7) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.454684 28455 tablet_service.cc:1514] Processing DeleteTablet for tablet 605531ed29214b1ca27011c5f5e96b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.454955 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f08aac9a7147ebaffda4262a57a86a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.456768 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 93462d2b1a474bd688d6266cd3b0695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.456749 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 129717ef137c41c5a08d401c64164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.457353 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 129717ef137c41c5a08d401c64164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.459405 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8097ba5606cf42d68448833de36d85c7) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.460728 28604 tablet_bootstrap.cc:492] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:12.461874 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8097ba5606cf42d68448833de36d85c7) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.462311 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]) successfully deleted
W20240503 14:12:12.462622 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 00b52ea336524beea2f255c8720f7472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00b52ea336524beea2f255c8720f7472 already in progress: creating tablet
I20240503 14:12:12.463409 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.462837 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e9609112657f427cbeeb3ef8f28cddf0 (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]) successfully deleted
W20240503 14:12:12.463899 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0982b99a02ee48f2b6379e7be56c4c02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0982b99a02ee48f2b6379e7be56c4c02 already in progress: creating tablet
I20240503 14:12:12.463634 28604 tablet_bootstrap.cc:654] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.464555 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6371e68ee85a47cdb0e8e21f7124d94a (table default.Table_355c3b14ae4f480dba47c2c90dc9dba2 [id=d7eac407621f4b03b21439fc2b0f7dc3]) successfully deleted
I20240503 14:12:12.447510 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 403) took 1767 ms. Trace:
I20240503 14:12:12.465719 28603 rpcz_store.cc:276] 0503 14:12:10.679581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.681515 (+ 1934us) service_pool.cc:225] Handling call
0503 14:12:12.447293 (+1765778us) ts_tablet_manager.cc:1120] Deleting tablet 646f83185d5241268d4126d10561e937
0503 14:12:12.447304 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.447424 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1765513}
I20240503 14:12:12.466341 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 402) took 1787 ms. Trace:
I20240503 14:12:12.466462 28603 rpcz_store.cc:276] 0503 14:12:10.679228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.687430 (+ 8202us) service_pool.cc:225] Handling call
0503 14:12:12.466049 (+1778619us) ts_tablet_manager.cc:1120] Deleting tablet 4e8411f474c247c2abfd0bd9682fb8a2
0503 14:12:12.466064 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.466242 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1778346}
I20240503 14:12:12.467115 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 405) took 1754 ms. Trace:
I20240503 14:12:12.467237 28603 rpcz_store.cc:276] 0503 14:12:10.712455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.713047 (+ 592us) service_pool.cc:225] Handling call
0503 14:12:12.466897 (+1753850us) ts_tablet_manager.cc:1120] Deleting tablet 49445f0f1dd1451c9ba066e61273f2f1
0503 14:12:12.466907 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.467019 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1752101}
I20240503 14:12:12.467677 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 407) took 1674 ms. Trace:
I20240503 14:12:12.467876 28653 ts_tablet_manager.cc:1875] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.467784 28603 rpcz_store.cc:276] 0503 14:12:10.793351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.793418 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:12.467465 (+1674047us) ts_tablet_manager.cc:1120] Deleting tablet 6696a237c3dd428097e2ebdf24afd63f
0503 14:12:12.467474 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.467592 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673721}
W20240503 14:12:12.467998 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b24ab3a5bba04a5092225a981f292222 consensus state has no leader
I20240503 14:12:12.468271 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 408) took 1674 ms. Trace:
I20240503 14:12:12.468382 28603 rpcz_store.cc:276] 0503 14:12:10.794017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.794327 (+ 310us) service_pool.cc:225] Handling call
0503 14:12:12.468056 (+1673729us) ts_tablet_manager.cc:1120] Deleting tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd
0503 14:12:12.468064 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.468190 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673485}
I20240503 14:12:12.468786 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 406) took 1675 ms. Trace:
I20240503 14:12:12.468887 28603 rpcz_store.cc:276] 0503 14:12:10.793098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.795621 (+ 2523us) service_pool.cc:225] Handling call
0503 14:12:12.468587 (+1672966us) ts_tablet_manager.cc:1120] Deleting tablet d6f5103b31ed45b480252dcf94b72f9c
0503 14:12:12.468595 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.468707 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672647}
I20240503 14:12:12.469261 28603 tablet_replica.cc:330] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.469488 28603 raft_consensus.cc:2229] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.469743 28603 raft_consensus.cc:2258] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.470391 28603 ts_tablet_manager.cc:1875] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:12.474210 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 646f83185d5241268d4126d10561e937 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 646f83185d5241268d4126d10561e937 already in progress: creating tablet
W20240503 14:12:12.474996 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 4e8411f474c247c2abfd0bd9682fb8a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e8411f474c247c2abfd0bd9682fb8a2 already in progress: creating tablet
W20240503 14:12:12.475368 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 49445f0f1dd1451c9ba066e61273f2f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49445f0f1dd1451c9ba066e61273f2f1 already in progress: creating tablet
W20240503 14:12:12.475663 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6696a237c3dd428097e2ebdf24afd63f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6696a237c3dd428097e2ebdf24afd63f already in progress: creating tablet
W20240503 14:12:12.475952 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd already in progress: creating tablet
W20240503 14:12:12.476238 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d6f5103b31ed45b480252dcf94b72f9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6f5103b31ed45b480252dcf94b72f9c already in progress: creating tablet
I20240503 14:12:12.480777 28024 catalog_manager.cc:5484] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:12.481312 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:12.484470 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f08aac9a7147ebaffda4262a57a86a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.485996 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3703ab7db61b4c9ead3058bfdc123189" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.486847 29330 raft_consensus.cc:482] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.487089 29330 raft_consensus.cc:504] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.487488 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 6809b59e87644316b9a450d6b670da2b (DEFAULT_TABLE table=default.Table_40fb6c7d8a074d12962a5f9b17b3c14f [id=88def62c7cf54150bc60da5cb6ab78a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.487999 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6809b59e87644316b9a450d6b670da2b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:12.489703 28150 leader_election.cc:341] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:12.490173 28150 leader_election.cc:341] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.490810 29281 raft_consensus.cc:482] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.490937 29330 leader_election.cc:288] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.490984 29281 raft_consensus.cc:504] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.491240 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f08aac9a7147ebaffda4262a57a86a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.491811 29281 leader_election.cc:288] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:12.494511 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3703ab7db61b4c9ead3058bfdc123189" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.496994 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 6809b59e87644316b9a450d6b670da2b (DEFAULT_TABLE table=default.Table_40fb6c7d8a074d12962a5f9b17b3c14f [id=88def62c7cf54150bc60da5cb6ab78a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.497023 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.497658 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6809b59e87644316b9a450d6b670da2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.497782 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 (DEFAULT_TABLE table=default.Table_9ecfb667ad96405c894e48eda793fada [id=5629316959054b54a63bac11c6d3d8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.498255 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.500072 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 6809b59e87644316b9a450d6b670da2b (DEFAULT_TABLE table=default.Table_40fb6c7d8a074d12962a5f9b17b3c14f [id=88def62c7cf54150bc60da5cb6ab78a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.500561 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6809b59e87644316b9a450d6b670da2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.505731 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 218481d1ddd246cf9aa9c0e163666f4e (DEFAULT_TABLE table=default.Table_1c2668b8ddb741298f29976408a1caee [id=4bfb3a5cb63f4af2a4abbab25f2306b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.506498 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218481d1ddd246cf9aa9c0e163666f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.507526 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651478ef632144079f93f3b50bff4611" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:12.507969 28215 raft_consensus.cc:2379] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:12.497023 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 (DEFAULT_TABLE table=default.Table_9ecfb667ad96405c894e48eda793fada [id=5629316959054b54a63bac11c6d3d8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.508803 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.508702 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf863dabf6647618732a03af84efab3 (DEFAULT_TABLE table=default.Table_7b4a078c66b0477180bed5b7b139de58 [id=e47d9e4b4933422e962efd3117aea50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.509322 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf863dabf6647618732a03af84efab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.511482 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6932067dd9e648e0a368cea1aae2eaa3" } modify_external_catalogs: true
I20240503 14:12:12.511855 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_46ee005206eb49879a8df1d11d8d331c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.512728 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 (DEFAULT_TABLE table=default.Table_9ecfb667ad96405c894e48eda793fada [id=5629316959054b54a63bac11c6d3d8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.513204 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.514465 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 79d09bee02c14822aa08050eac5be181 (DEFAULT_TABLE table=default.Table_c12ba1b49a3f4f509ff427bbeb4723af [id=9c7309de673b4590a2d02e6f2519103c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.515079 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d09bee02c14822aa08050eac5be181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.515527 28460 tablet_service.cc:1467] Processing CreateTablet for tablet d5d7687bbd9c41e49f2776019683a32f (DEFAULT_TABLE table=default.Table_c383582e14364f50a46da3692bedd880 [id=8956c29cbf9446708bcfe3821f6907ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.515635 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_38aab413057643f78d942ab06e5905f7" } new_table_name: "default.Table_34299e0857054fa5bc1b9b671c0842ba" modify_external_catalogs: true
I20240503 14:12:12.516011 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d7687bbd9c41e49f2776019683a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.517036 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_80d78826082349c882ea2c0ca66e0141" } modify_external_catalogs: true
I20240503 14:12:12.517272 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_80d78826082349c882ea2c0ca66e0141" } modify_external_catalogs: true
I20240503 14:12:12.518757 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6932067dd9e648e0a368cea1aae2eaa3" } modify_external_catalogs: true
I20240503 14:12:12.519407 28179 tablet_service.cc:1467] Processing CreateTablet for tablet a6b468dd4102495abd4337edcfc6340a (DEFAULT_TABLE table=default.Table_7efeff78f7b04eadbc81cf4b5fb025a1 [id=ad1eccda9b8b4c0d8d6da1544c583be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.519940 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b468dd4102495abd4337edcfc6340a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:12.519994 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46ee005206eb49879a8df1d11d8d331c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.521149 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a0200d3cb5e84132adf59bc5c3241d88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.521719 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0200d3cb5e84132adf59bc5c3241d88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.524488 28462 tablet_service.cc:1467] Processing CreateTablet for tablet a6b468dd4102495abd4337edcfc6340a (DEFAULT_TABLE table=default.Table_7efeff78f7b04eadbc81cf4b5fb025a1 [id=ad1eccda9b8b4c0d8d6da1544c583be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.525094 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b468dd4102495abd4337edcfc6340a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.527834 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d5808426f2954b4981e680e8a56f5a5b" } new_table_name: "default.Table_dffe0d8805804b6a8c235ba8f8772152" modify_external_catalogs: true
I20240503 14:12:12.529539 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2795914729c948ed94f38a47c734d74d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.530125 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2795914729c948ed94f38a47c734d74d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.530139 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf863dabf6647618732a03af84efab3 (DEFAULT_TABLE table=default.Table_7b4a078c66b0477180bed5b7b139de58 [id=e47d9e4b4933422e962efd3117aea50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.530614 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf863dabf6647618732a03af84efab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.531147 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 218481d1ddd246cf9aa9c0e163666f4e (DEFAULT_TABLE table=default.Table_1c2668b8ddb741298f29976408a1caee [id=4bfb3a5cb63f4af2a4abbab25f2306b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.531714 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218481d1ddd246cf9aa9c0e163666f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.532151 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 79d09bee02c14822aa08050eac5be181 (DEFAULT_TABLE table=default.Table_c12ba1b49a3f4f509ff427bbeb4723af [id=9c7309de673b4590a2d02e6f2519103c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.532853 29317 raft_consensus.cc:482] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.532949 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d09bee02c14822aa08050eac5be181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.533064 29317 raft_consensus.cc:504] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.533782 29317 leader_election.cc:288] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.534579 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a5bf2e0d9bcc43b0bec8b4daf11b707b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.535243 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5bf2e0d9bcc43b0bec8b4daf11b707b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:12.535960 28427 leader_election.cc:341] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0b255d3e3dc64fb69c2ef6ed389db6a6
I20240503 14:12:12.536393 28333 tablet_service.cc:1467] Processing CreateTablet for tablet a6b468dd4102495abd4337edcfc6340a (DEFAULT_TABLE table=default.Table_7efeff78f7b04eadbc81cf4b5fb025a1 [id=ad1eccda9b8b4c0d8d6da1544c583be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.538228 28460 tablet_service.cc:1467] Processing CreateTablet for tablet a6c17cfce7e644668817bf3483a098d4 (DEFAULT_TABLE table=default.Table_c8a1b23185cf4c51ac94cb268e9e2222 [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.538498 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 007a892ef8504b848a400c05740eaa0c
I20240503 14:12:12.538684 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c17cfce7e644668817bf3483a098d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.538985 28028 catalog_manager.cc:6554] ReplaceTablet: tablet 007a892ef8504b848a400c05740eaa0c deleted and replaced by tablet 320c526e845e4dc78da345bcb095918f
I20240503 14:12:12.539429 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7aefabd0c3841618e68382b8fa41a9a" } modify_external_catalogs: true
I20240503 14:12:12.539644 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7aefabd0c3841618e68382b8fa41a9a" } modify_external_catalogs: true
I20240503 14:12:12.540675 28661 ts_tablet_manager.cc:1888] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:12.540861 28661 log.cc:1196] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/da19787034c24ebe929601d49cb21b3b
I20240503 14:12:12.541385 28661 ts_tablet_manager.cc:1909] T da19787034c24ebe929601d49cb21b3b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:12.542202 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_98de9a28340945fc977e41d46d3361c0" } new_table_name: "default.Table_d201fddd164546b5a2d529d6bc79cb36" modify_external_catalogs: true
I20240503 14:12:12.544054 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b468dd4102495abd4337edcfc6340a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.544896 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.528379 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 218481d1ddd246cf9aa9c0e163666f4e (DEFAULT_TABLE table=default.Table_1c2668b8ddb741298f29976408a1caee [id=4bfb3a5cb63f4af2a4abbab25f2306b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.548661 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 79d09bee02c14822aa08050eac5be181 (DEFAULT_TABLE table=default.Table_c12ba1b49a3f4f509ff427bbeb4723af [id=9c7309de673b4590a2d02e6f2519103c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.548800 28196 tablet_service.cc:1467] Processing CreateTablet for tablet a6c17cfce7e644668817bf3483a098d4 (DEFAULT_TABLE table=default.Table_c8a1b23185cf4c51ac94cb268e9e2222 [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.549393 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c17cfce7e644668817bf3483a098d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.549765 28180 tablet_service.cc:1467] Processing CreateTablet for tablet d5d7687bbd9c41e49f2776019683a32f (DEFAULT_TABLE table=default.Table_c383582e14364f50a46da3692bedd880 [id=8956c29cbf9446708bcfe3821f6907ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.550215 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d7687bbd9c41e49f2776019683a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.551402 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218481d1ddd246cf9aa9c0e163666f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.551471 28290 leader_election.cc:302] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.552701 29330 raft_consensus.cc:3037] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:12.554433 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.555696 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf863dabf6647618732a03af84efab3 (DEFAULT_TABLE table=default.Table_7b4a078c66b0477180bed5b7b139de58 [id=e47d9e4b4933422e962efd3117aea50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.555282 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.556600 28325 tablet_service.cc:1467] Processing CreateTablet for tablet a6c17cfce7e644668817bf3483a098d4 (DEFAULT_TABLE table=default.Table_c8a1b23185cf4c51ac94cb268e9e2222 [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.557044 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c17cfce7e644668817bf3483a098d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.557457 28465 tablet_service.cc:1467] Processing CreateTablet for tablet d9c7b5fb9b4a483c8292f68387591034 (DEFAULT_TABLE table=default.Table_44fe718f4ee1446bba6f7cd746b0154a [id=39dbc007b7a5429a98e1cfa1af884c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.557837 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.560376 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b255d3e3dc64fb69c2ef6ed389db6a6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.561491 28600 tablet_bootstrap.cc:492] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:12.562006 28600 ts_tablet_manager.cc:1367] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.404s user 0.007s sys 0.006s
W20240503 14:12:12.562278 28150 leader_election.cc:341] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.558341 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d09bee02c14822aa08050eac5be181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.564301 28322 tablet_service.cc:1467] Processing CreateTablet for tablet d9c7b5fb9b4a483c8292f68387591034 (DEFAULT_TABLE table=default.Table_44fe718f4ee1446bba6f7cd746b0154a [id=39dbc007b7a5429a98e1cfa1af884c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.564869 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c7b5fb9b4a483c8292f68387591034. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:12.566879 28427 leader_election.cc:341] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.567104 28427 leader_election.cc:302] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:12.567739 29364 raft_consensus.cc:2726] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.561873 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b139ab0b5864024ac48215d46b0a7bd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.566689 28336 tablet_service.cc:1467] Processing CreateTablet for tablet d5d7687bbd9c41e49f2776019683a32f (DEFAULT_TABLE table=default.Table_c383582e14364f50a46da3692bedd880 [id=8956c29cbf9446708bcfe3821f6907ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.572484 28318 tablet_service.cc:1467] Processing CreateTablet for tablet f080c95ff5ca41e0b3d27382040107e9 (DEFAULT_TABLE table=default.Table_e446d24cec95467d98278762338e11d7 [id=9eb8c916fc3c435497209c58ca5b32a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.573925 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b139ab0b5864024ac48215d46b0a7bd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.574506 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f080c95ff5ca41e0b3d27382040107e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.574506 28193 tablet_service.cc:1467] Processing CreateTablet for tablet f080c95ff5ca41e0b3d27382040107e9 (DEFAULT_TABLE table=default.Table_e446d24cec95467d98278762338e11d7 [id=9eb8c916fc3c435497209c58ca5b32a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.575966 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d7687bbd9c41e49f2776019683a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.580504 28150 leader_election.cc:302] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:12.580734 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e7b2865e9604ce58280321c7ee2a88e consensus state has no leader
I20240503 14:12:12.581260 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_afc737ad630d4de6b239f807771fa6a1" } new_table_name: "default.Table_8b6756d9ceab442784c7aa1436e25032" modify_external_catalogs: true
W20240503 14:12:12.582432 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e45df464f6954a56bec8b107b350320c consensus state has no leader
I20240503 14:12:12.582906 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a1449d50468849dcbb0b89dac2774255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.583508 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf863dabf6647618732a03af84efab3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:12.583832 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1449d50468849dcbb0b89dac2774255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.584719 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_33b986666be7452c8edcd5b0a0bf4f6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.585175 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b476e87faea1415d95ae014a93fa22af
W20240503 14:12:12.585270 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33b986666be7452c8edcd5b0a0bf4f6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.585383 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f080c95ff5ca41e0b3d27382040107e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.586738 28182 tablet_service.cc:1467] Processing CreateTablet for tablet d9c7b5fb9b4a483c8292f68387591034 (DEFAULT_TABLE table=default.Table_44fe718f4ee1446bba6f7cd746b0154a [id=39dbc007b7a5429a98e1cfa1af884c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.588483 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d4c8a97076a44f9eaa3a15026f90a304
W20240503 14:12:12.588485 28150 leader_election.cc:341] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.588721 28600 raft_consensus.cc:348] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.589038 28600 raft_consensus.cc:374] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.589402 28600 raft_consensus.cc:729] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.589740 28600 consensus_queue.cc:260] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.590968 28600 ts_tablet_manager.cc:1398] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240503 14:12:12.591225 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c7b5fb9b4a483c8292f68387591034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.591146 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4fdbe31ba57a42b993d60fa19081c044" } modify_external_catalogs: true
I20240503 14:12:12.591394 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4fdbe31ba57a42b993d60fa19081c044" } modify_external_catalogs: true
I20240503 14:12:12.592027 29330 raft_consensus.cc:2726] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.597671 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e87faea1415d95ae014a93fa22af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.598546 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c8a97076a44f9eaa3a15026f90a304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.603583 28600 tablet_bootstrap.cc:492] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:12.607074 28614 tablet_bootstrap.cc:492] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:12.607337 28614 ts_tablet_manager.cc:1367] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.416s user 0.037s sys 0.049s
I20240503 14:12:12.608430 28614 raft_consensus.cc:348] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.608711 28614 raft_consensus.cc:374] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.608824 28614 raft_consensus.cc:729] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.609185 28614 consensus_queue.cc:260] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.609865 28614 ts_tablet_manager.cc:1398] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:12.611990 28614 tablet_bootstrap.cc:492] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:12.617079 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c7b5fb9b4a483c8292f68387591034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.617712 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:12.618947 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3a0134977dde48a0849c29c36487c742" } modify_external_catalogs: true
I20240503 14:12:12.619284 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3a0134977dde48a0849c29c36487c742" } modify_external_catalogs: true
I20240503 14:12:12.619341 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 352) took 2424 ms. Trace:
I20240503 14:12:12.619663 28463 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e87faea1415d95ae014a93fa22af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.620016 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c8a97076a44f9eaa3a15026f90a304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.623526 28458 tablet_service.cc:1467] Processing CreateTablet for tablet f080c95ff5ca41e0b3d27382040107e9 (DEFAULT_TABLE table=default.Table_e446d24cec95467d98278762338e11d7 [id=9eb8c916fc3c435497209c58ca5b32a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.624076 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f080c95ff5ca41e0b3d27382040107e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.625030 28614 tablet_bootstrap.cc:654] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.624241 28661 rpcz_store.cc:276] 0503 14:12:10.193339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.193421 (+ 82us) service_pool.cc:225] Handling call
0503 14:12:12.181558 (+1988137us) ts_tablet_manager.cc:1120] Deleting tablet da19787034c24ebe929601d49cb21b3b
0503 14:12:12.181571 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.199837 (+ 18266us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.540521 (+340684us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.617581 (+ 77060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1987904}
W20240503 14:12:12.627977 29295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:12.629962 29295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:12.630206 29295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:12.633265 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:12.633738 28150 leader_election.cc:302] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.634586 29382 raft_consensus.cc:2726] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.628397 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 353) took 2433 ms. Trace:
I20240503 14:12:12.637217 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet da19787034c24ebe929601d49cb21b3b (table default.Table_6c996bea4ba44fefbb3bc602c3c76f58 [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:12.638689 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_251eaa8fc8ff4652b02caa68687f4ec0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.638870 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_90131726ecc0494f8e4eff1c20c6095f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.639349 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_251eaa8fc8ff4652b02caa68687f4ec0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:12.639576 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90131726ecc0494f8e4eff1c20c6095f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.641144 28661 rpcz_store.cc:276] 0503 14:12:10.194135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.194237 (+ 102us) service_pool.cc:225] Handling call
0503 14:12:12.625605 (+2431368us) ts_tablet_manager.cc:1120] Deleting tablet 95d9985eecda4f3194c3251216aa58ee
0503 14:12:12.625624 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.627928 (+ 2304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2431162}
I20240503 14:12:12.642952 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 349) took 2453 ms. Trace:
I20240503 14:12:12.643095 28661 rpcz_store.cc:276] 0503 14:12:10.188296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.188387 (+ 91us) service_pool.cc:225] Handling call
0503 14:12:12.641809 (+2453422us) ts_tablet_manager.cc:1120] Deleting tablet 0b139ab0b5864024ac48215d46b0a7bd
0503 14:12:12.641822 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.641970 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2446741}
I20240503 14:12:12.648331 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c8a97076a44f9eaa3a15026f90a304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.648973 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e87faea1415d95ae014a93fa22af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.649061 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 354) took 2395 ms. Trace:
I20240503 14:12:12.649236 28661 rpcz_store.cc:276] 0503 14:12:10.248107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.248342 (+ 235us) service_pool.cc:225] Handling call
0503 14:12:12.643574 (+2395232us) ts_tablet_manager.cc:1120] Deleting tablet a884cf4ea11e48ef941722b79ad50075
0503 14:12:12.643585 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.643745 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2296839}
I20240503 14:12:12.650045 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 364) took 2260 ms. Trace:
I20240503 14:12:12.650167 28661 rpcz_store.cc:276] 0503 14:12:10.389849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.389945 (+ 96us) service_pool.cc:225] Handling call
0503 14:12:12.649708 (+2259763us) ts_tablet_manager.cc:1120] Deleting tablet 02d45788ae644a8581acca54cc55dd3f
0503 14:12:12.649720 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.649849 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2258022}
I20240503 14:12:12.652714 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 365) took 2259 ms. Trace:
W20240503 14:12:12.652767 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 71108us
I20240503 14:12:12.652992 28661 rpcz_store.cc:276] 0503 14:12:10.393700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.393779 (+ 79us) service_pool.cc:225] Handling call
0503 14:12:12.650600 (+2256821us) ts_tablet_manager.cc:1120] Deleting tablet 3d54cc5c03034a6685c1da34f9bd65ac
0503 14:12:12.650609 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.652607 (+ 1998us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2255951}
I20240503 14:12:12.654021 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 366) took 2252 ms. Trace:
I20240503 14:12:12.654227 28661 rpcz_store.cc:276] 0503 14:12:10.401719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.401870 (+ 151us) service_pool.cc:225] Handling call
0503 14:12:12.653783 (+2251913us) ts_tablet_manager.cc:1120] Deleting tablet 0bd5234791524eddb7edd493e39fb993
0503 14:12:12.653793 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.653910 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2249382}
I20240503 14:12:12.654858 28661 tablet_replica.cc:330] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:12.655005 29382 raft_consensus.cc:2726] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.655143 28661 raft_consensus.cc:2229] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:12.657335 29295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:12.657586 29295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:12.657761 29295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:12.657945 29295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240503 14:12:12.665228 28150 leader_election.cc:341] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.668929 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7b6c998e518541248a87e4c1551ec080
W20240503 14:12:12.670046 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 780b53efefcf42569bba2fcabd21e211 consensus state has no leader
I20240503 14:12:12.661841 28604 tablet_bootstrap.cc:492] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:12.671511 28604 ts_tablet_manager.cc:1367] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.466s user 0.010s sys 0.015s
I20240503 14:12:12.671553 28661 raft_consensus.cc:2258] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.671634 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_334a9c3578d445778df17229f9d401f7" } new_table_name: "default.Table_48d4c9741adf4c52b6c0732c56987255" modify_external_catalogs: true
I20240503 14:12:12.672592 28604 raft_consensus.cc:348] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.672845 28604 raft_consensus.cc:374] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.672991 28604 raft_consensus.cc:729] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.673329 28661 ts_tablet_manager.cc:1875] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.673283 28604 consensus_queue.cc:260] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.673288 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ec3edf11cc5746799ddb3623736b23fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.673900 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec3edf11cc5746799ddb3623736b23fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.673945 28604 ts_tablet_manager.cc:1398] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:12.674357 28604 tablet_bootstrap.cc:492] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:12.677192 28604 tablet_bootstrap.cc:654] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.685945 29295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.253:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.50.253
--webserver_port=43185
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:12.686949 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3d53a12202f9477788f519d5ef0c9531"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.687678 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d53a12202f9477788f519d5ef0c9531 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.694588 29295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:12.697258 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ce4846d662a64d51b16fd5447c371e9a
I20240503 14:12:12.697355 29295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:12.699330 28661 ts_tablet_manager.cc:1888] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.699653 28661 log.cc:1196] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/93462d2b1a474bd688d6266cd3b0695c
I20240503 14:12:12.705214 29330 raft_consensus.cc:482] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.705437 29330 raft_consensus.cc:504] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:12.706198 28150 leader_election.cc:341] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.706274 29330 leader_election.cc:288] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.707975 29330 consensus_queue.cc:1035] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:12.708868 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 43639.2us
I20240503 14:12:12.707970 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93462d2b1a474bd688d6266cd3b0695c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.711977 28661 ts_tablet_manager.cc:1909] T 93462d2b1a474bd688d6266cd3b0695c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:12.712306 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c7c5dfac02d4dfe932996d2a37fe031 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:12.712610 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 95d9985eecda4f3194c3251216aa58ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95d9985eecda4f3194c3251216aa58ee already in progress: creating tablet
W20240503 14:12:12.713038 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0b139ab0b5864024ac48215d46b0a7bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b139ab0b5864024ac48215d46b0a7bd already in progress: creating tablet
I20240503 14:12:12.713104 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 378) took 2180 ms. Trace:
I20240503 14:12:12.713348 28661 rpcz_store.cc:276] 0503 14:12:10.532367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.532428 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:12.654745 (+2122317us) ts_tablet_manager.cc:1120] Deleting tablet 93462d2b1a474bd688d6266cd3b0695c
0503 14:12:12.654754 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.694863 (+ 40109us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.699154 (+ 4291us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.713046 (+ 13892us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2065171}
W20240503 14:12:12.713507 28290 leader_election.cc:341] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:12.714519 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 381) took 2181 ms. Trace:
I20240503 14:12:12.714768 28661 rpcz_store.cc:276] 0503 14:12:10.532754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.762073 (+229319us) service_pool.cc:225] Handling call
0503 14:12:12.714038 (+1951965us) ts_tablet_manager.cc:1120] Deleting tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
0503 14:12:12.714052 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.714338 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1912260}
I20240503 14:12:12.715638 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 382) took 2182 ms. Trace:
W20240503 14:12:12.715792 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a884cf4ea11e48ef941722b79ad50075 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a884cf4ea11e48ef941722b79ad50075 already in progress: creating tablet
I20240503 14:12:12.715873 28661 rpcz_store.cc:276] 0503 14:12:10.532838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.785034 (+252196us) service_pool.cc:225] Handling call
0503 14:12:12.715375 (+1930341us) ts_tablet_manager.cc:1120] Deleting tablet 8c6202333af943cbbc0d09553c1c193c
0503 14:12:12.715385 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.715567 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1913364}
W20240503 14:12:12.716153 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 02d45788ae644a8581acca54cc55dd3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d45788ae644a8581acca54cc55dd3f already in progress: creating tablet
W20240503 14:12:12.716517 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 3d54cc5c03034a6685c1da34f9bd65ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d54cc5c03034a6685c1da34f9bd65ac already in progress: creating tablet
W20240503 14:12:12.716735 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0bd5234791524eddb7edd493e39fb993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bd5234791524eddb7edd493e39fb993 already in progress: creating tablet
I20240503 14:12:12.716737 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 383) took 2183 ms. Trace:
I20240503 14:12:12.716948 28661 rpcz_store.cc:276] 0503 14:12:10.532922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.787732 (+254810us) service_pool.cc:225] Handling call
0503 14:12:12.716498 (+1928766us) ts_tablet_manager.cc:1120] Deleting tablet a035a5249f8348b591b27ba3fa7ae43a
0503 14:12:12.716507 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.716676 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1913769}
I20240503 14:12:12.717247 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 93462d2b1a474bd688d6266cd3b0695c (table default.Table_7712b0c57d054073ac3f31fa183e72f1 [id=48c3f3e039044d10a6cfa1b6bcd5ae79]) successfully deleted
W20240503 14:12:12.717468 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a already in progress: creating tablet
W20240503 14:12:12.717691 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 8c6202333af943cbbc0d09553c1c193c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6202333af943cbbc0d09553c1c193c already in progress: creating tablet
I20240503 14:12:12.717725 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 387) took 2132 ms. Trace:
W20240503 14:12:12.717919 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a035a5249f8348b591b27ba3fa7ae43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a035a5249f8348b591b27ba3fa7ae43a already in progress: creating tablet
I20240503 14:12:12.717960 28661 rpcz_store.cc:276] 0503 14:12:10.585675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.802753 (+217078us) service_pool.cc:225] Handling call
0503 14:12:12.717471 (+1914718us) ts_tablet_manager.cc:1120] Deleting tablet 20b176a071ba4f3fbff2a64d6e46027a
0503 14:12:12.717482 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.717661 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849199}
I20240503 14:12:12.718873 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6c998e518541248a87e4c1551ec080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.719063 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044f34588d045ce898435b61a90503f with delete_type TABLET_DATA_DELETED (replaced by tablet daf5d153ab5142c0902cc6e96629f412) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.719480 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6c998e518541248a87e4c1551ec080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.719939 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 386) took 2139 ms. Trace:
I20240503 14:12:12.720163 28661 rpcz_store.cc:276] 0503 14:12:10.580220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.802030 (+221810us) service_pool.cc:225] Handling call
0503 14:12:12.719764 (+1917734us) ts_tablet_manager.cc:1120] Deleting tablet 0982b99a02ee48f2b6379e7be56c4c02
0503 14:12:12.719774 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.719881 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1814754}
I20240503 14:12:12.720810 28661 tablet_replica.cc:330] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:12.721128 28661 raft_consensus.cc:2229] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.721515 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.721472 28150 leader_election.cc:302] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.722524 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4846d662a64d51b16fd5447c371e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:12.723199 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4846d662a64d51b16fd5447c371e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:12.723508 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6c998e518541248a87e4c1551ec080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:12.724437 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 20b176a071ba4f3fbff2a64d6e46027a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20b176a071ba4f3fbff2a64d6e46027a already in progress: creating tablet
W20240503 14:12:12.724670 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0982b99a02ee48f2b6379e7be56c4c02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0982b99a02ee48f2b6379e7be56c4c02 already in progress: creating tablet
I20240503 14:12:12.728639 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet 495190fca06a40a386c8215eb2195920 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:12.729786 28603 ts_tablet_manager.cc:1888] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.731820 28614 tablet_bootstrap.cc:492] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:12.734515 28614 ts_tablet_manager.cc:1367] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.007s
I20240503 14:12:12.735674 28614 raft_consensus.cc:348] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.736027 28614 raft_consensus.cc:374] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.736248 28614 raft_consensus.cc:729] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.736642 28614 consensus_queue.cc:260] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:12.737603 28604 tablet_bootstrap.cc:492] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:12.737879 28604 ts_tablet_manager.cc:1367] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.064s user 0.020s sys 0.007s
I20240503 14:12:12.738842 28614 ts_tablet_manager.cc:1398] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:12.739002 28604 raft_consensus.cc:348] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.739215 28614 tablet_bootstrap.cc:492] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:12.739395 28604 raft_consensus.cc:374] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.739554 28604 raft_consensus.cc:729] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.739895 28604 consensus_queue.cc:260] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.740823 28604 ts_tablet_manager.cc:1398] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:12.741286 28603 log.cc:1196] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/74df7781cc6a4dd6b6fd78a728ffd71b
I20240503 14:12:12.741849 28603 ts_tablet_manager.cc:1909] T 74df7781cc6a4dd6b6fd78a728ffd71b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.742305 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 420) took 1437 ms. Trace:
I20240503 14:12:12.743830 28603 rpcz_store.cc:276] 0503 14:12:11.305066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.313712 (+ 8646us) service_pool.cc:225] Handling call
0503 14:12:12.469133 (+1155421us) ts_tablet_manager.cc:1120] Deleting tablet 74df7781cc6a4dd6b6fd78a728ffd71b
0503 14:12:12.469141 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.625482 (+156341us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.729608 (+104126us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.742254 (+ 12646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104317}
I20240503 14:12:12.745151 28604 tablet_bootstrap.cc:492] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:12.746594 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 422) took 1441 ms. Trace:
I20240503 14:12:12.746824 28603 rpcz_store.cc:276] 0503 14:12:11.305398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.306043 (+ 645us) service_pool.cc:225] Handling call
0503 14:12:12.744332 (+1438289us) ts_tablet_manager.cc:1120] Deleting tablet 8b1728b6f0e64661aaa8184ce51366d2
0503 14:12:12.746355 (+ 2023us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.746536 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358121}
I20240503 14:12:12.746996 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 74df7781cc6a4dd6b6fd78a728ffd71b (table default.Table_fc189ad8c19041f7814fac6de9f482cb [id=b7f4f49135e245dc975ffd7eb0589a8d]) successfully deleted
I20240503 14:12:12.743687 28614 tablet_bootstrap.cc:654] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.749758 28604 tablet_bootstrap.cc:654] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:12.756574 29399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:12.756812 28290 leader_election.cc:341] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 93462d2b1a474bd688d6266cd3b0695c
I20240503 14:12:12.757002 28290 leader_election.cc:302] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.757520 29330 raft_consensus.cc:2726] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.758697 29382 raft_consensus.cc:2726] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:12.758899 29397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:12.763460 28600 tablet_bootstrap.cc:654] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.763437 29281 raft_consensus.cc:482] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:12.765784 29396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:12.766395 29281 raft_consensus.cc:504] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.767242 29281 leader_election.cc:288] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:12.767943 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dc7d96357743089372baf71654b1e3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.768033 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4846d662a64d51b16fd5447c371e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:12.769413 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dc7d96357743089372baf71654b1e3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:12.770737 28150 leader_election.cc:341] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:12.771485 28150 leader_election.cc:341] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.771642 28150 leader_election.cc:302] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.772330 29382 raft_consensus.cc:2726] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.774340 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:12.775858 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 50edc38cdbe74397aa4f4928d56d9fbc from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:12.785856 28653 ts_tablet_manager.cc:1888] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:12.771353 28661 raft_consensus.cc:2258] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.787025 28653 log.cc:1196] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e9609112657f427cbeeb3ef8f28cddf0
I20240503 14:12:12.789342 28653 ts_tablet_manager.cc:1909] T e9609112657f427cbeeb3ef8f28cddf0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:12.790110 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c6f1f5b922f549679f50474fd6412cfc" } new_table_name: "default.Table_25270ebff04b49058519a05acc90c215" modify_external_catalogs: true
I20240503 14:12:12.791678 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c89fe31dfbac4d48a061a21a358b84f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.792332 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c89fe31dfbac4d48a061a21a358b84f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.792527 28600 tablet_bootstrap.cc:492] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:12.792774 28600 ts_tablet_manager.cc:1367] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.200s user 0.007s sys 0.005s
I20240503 14:12:12.793713 28600 raft_consensus.cc:348] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.794096 28600 raft_consensus.cc:374] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.794225 28600 raft_consensus.cc:729] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.794534 28600 consensus_queue.cc:260] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.788736 28661 ts_tablet_manager.cc:1875] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.796839 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 397) took 2293 ms. Trace:
I20240503 14:12:12.796945 28653 rpcz_store.cc:276] 0503 14:12:10.503703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.504117 (+ 414us) service_pool.cc:225] Handling call
0503 14:12:12.438778 (+1934661us) ts_tablet_manager.cc:1120] Deleting tablet e9609112657f427cbeeb3ef8f28cddf0
0503 14:12:12.438790 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.467725 (+ 28935us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x611000f53c70. stack: 00007f2f23940756 00007f2f23940510 00007f2f2297348d 0000000000955804 00000000009557cd 000000000095519a 00007f2f36de1f9d 00007f2f3035b05e 00007f2f30488fe7 00007f2f32f1c57c 00007f2f32f4e8d4 00007f2f32f2fbc0 00007f2f32f2f94d 0000000000a60bcc 00007f2f23986b74 00007f2f2398a19b
0503 14:12:12.472006 (+ 4281us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.785680 (+313674us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.796787 (+ 11107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59185536,"tablet-delete.queue_time_us":1934398}
I20240503 14:12:12.797441 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e9609112657f427cbeeb3ef8f28cddf0 (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]) successfully deleted
I20240503 14:12:12.798933 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 398) took 2293 ms. Trace:
I20240503 14:12:12.799037 28653 rpcz_store.cc:276] 0503 14:12:10.505699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.505756 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:12.798725 (+2292969us) ts_tablet_manager.cc:1120] Deleting tablet 00b52ea336524beea2f255c8720f7472
0503 14:12:12.798736 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.798875 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2292703}
W20240503 14:12:12.799566 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 00b52ea336524beea2f255c8720f7472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00b52ea336524beea2f255c8720f7472 already in progress: creating tablet
I20240503 14:12:12.800561 28653 tablet_replica.cc:330] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:12.800812 28653 raft_consensus.cc:2229] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.801101 28653 raft_consensus.cc:2258] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.801699 28653 ts_tablet_manager.cc:1875] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.808234 28603 tablet_replica.cc:330] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.809273 28603 raft_consensus.cc:2229] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.809638 28603 raft_consensus.cc:2258] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.810343 28603 ts_tablet_manager.cc:1875] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.823704 29382 raft_consensus.cc:482] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.823945 29382 raft_consensus.cc:504] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.825387 29364 raft_consensus.cc:482] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.825595 29364 raft_consensus.cc:504] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.826454 29364 leader_election.cc:288] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:12.826366 29330 consensus_queue.cc:1035] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:12.826478 28357 raft_consensus.cc:1261] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:12.827096 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265dc5f2630e453ea71abddb1a9753f3" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.827375 29364 consensus_queue.cc:1035] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:12.828824 28212 raft_consensus.cc:1261] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:12:12.829156 28427 leader_election.cc:341] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.830696 29364 consensus_queue.cc:1035] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:12.832442 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea261d8d5d174bee804aa2e171f35008 consensus state has no leader
I20240503 14:12:12.834380 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044f34588d045ce898435b61a90503f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.834956 29295 server_base.cc:1034] running on GCE node
I20240503 14:12:12.835151 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3169e805aa1543889ee0dab4758ae9e5" } new_table_name: "default.Table_a9e8bd96c6824e9c8c64036f0848c876" modify_external_catalogs: true
I20240503 14:12:12.836450 29295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:12.837388 28614 tablet_bootstrap.cc:492] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:12.837627 28614 ts_tablet_manager.cc:1367] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.002s
I20240503 14:12:12.838613 28614 raft_consensus.cc:348] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.838884 28614 raft_consensus.cc:374] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.838850 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044f34588d045ce898435b61a90503f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:12.838986 28614 raft_consensus.cc:729] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.839308 28614 consensus_queue.cc:260] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.839921 28614 ts_tablet_manager.cc:1398] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:12.842558 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a1449d50468849dcbb0b89dac2774255" } modify_external_catalogs: true
I20240503 14:12:12.842792 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a1449d50468849dcbb0b89dac2774255" } modify_external_catalogs: true
I20240503 14:12:12.842257 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2b685af443594f2f93b99eb4a1df3e13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.844666 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b685af443594f2f93b99eb4a1df3e13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.844739 28614 tablet_bootstrap.cc:492] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:12.850996 29382 leader_election.cc:288] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:12.852663 28150 leader_election.cc:341] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.852759 28614 tablet_bootstrap.cc:654] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:12.853775 28427 leader_election.cc:341] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 265dc5f2630e453ea71abddb1a9753f3
I20240503 14:12:12.853960 28427 leader_election.cc:302] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:12.854688 29317 raft_consensus.cc:2726] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.854893 28603 ts_tablet_manager.cc:1888] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:12.855067 28603 log.cc:1196] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/cae4969fc9474fdbaeae1b4d09e4875b
I20240503 14:12:12.855522 28603 ts_tablet_manager.cc:1909] T cae4969fc9474fdbaeae1b4d09e4875b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.855967 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 423) took 1550 ms. Trace:
I20240503 14:12:12.856151 28603 rpcz_store.cc:276] 0503 14:12:11.305545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.306181 (+ 636us) service_pool.cc:225] Handling call
0503 14:12:12.749290 (+1443109us) ts_tablet_manager.cc:1120] Deleting tablet cae4969fc9474fdbaeae1b4d09e4875b
0503 14:12:12.808132 (+ 58842us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.835494 (+ 27362us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.854751 (+ 19257us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.855909 (+ 1158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362453}
I20240503 14:12:12.856799 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet cae4969fc9474fdbaeae1b4d09e4875b (table default.Table_02a4e7ef8fdf45a5b0c1cea8a095b45f [id=837e8a35f3024ddcbe19763625d82a1f]) successfully deleted
I20240503 14:12:12.857473 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 421) took 1552 ms. Trace:
I20240503 14:12:12.857713 29382 raft_consensus.cc:482] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:12.857648 28603 rpcz_store.cc:276] 0503 14:12:11.305217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.306551 (+ 1334us) service_pool.cc:225] Handling call
0503 14:12:12.857224 (+1550673us) ts_tablet_manager.cc:1120] Deleting tablet a7295c2d3bd34e20a69986aadc9c7017
0503 14:12:12.857235 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.857375 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445331}
I20240503 14:12:12.857949 29382 raft_consensus.cc:504] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:12.858258 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a7295c2d3bd34e20a69986aadc9c7017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7295c2d3bd34e20a69986aadc9c7017 already in progress: creating tablet
I20240503 14:12:12.858769 29382 leader_election.cc:288] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:12.858865 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 424) took 1436 ms. Trace:
I20240503 14:12:12.858990 28603 rpcz_store.cc:276] 0503 14:12:11.422901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.423251 (+ 350us) service_pool.cc:225] Handling call
0503 14:12:12.858641 (+1435390us) ts_tablet_manager.cc:1120] Deleting tablet 0a50b3e199314bfc8465fdb7b2425d07
0503 14:12:12.858651 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.858774 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1423133}
W20240503 14:12:12.859647 28150 leader_election.cc:341] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.859885 28150 leader_election.cc:302] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:12.860222 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0a50b3e199314bfc8465fdb7b2425d07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a50b3e199314bfc8465fdb7b2425d07 already in progress: creating tablet
I20240503 14:12:12.860818 29382 raft_consensus.cc:2726] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:12.860859 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 425) took 1331 ms. Trace:
I20240503 14:12:12.861550 28603 rpcz_store.cc:276] 0503 14:12:11.530335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.530401 (+ 66us) service_pool.cc:225] Handling call
0503 14:12:12.860611 (+1330210us) ts_tablet_manager.cc:1120] Deleting tablet dbcc6cef7b7d464a9fbbae61caf0437d
0503 14:12:12.860621 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.860780 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329869}
W20240503 14:12:12.862576 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet dbcc6cef7b7d464a9fbbae61caf0437d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbcc6cef7b7d464a9fbbae61caf0437d already in progress: creating tablet
I20240503 14:12:12.863319 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 426) took 1330 ms. Trace:
I20240503 14:12:12.863904 28603 rpcz_store.cc:276] 0503 14:12:11.535321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.544689 (+ 9368us) service_pool.cc:225] Handling call
0503 14:12:12.863068 (+1318379us) ts_tablet_manager.cc:1120] Deleting tablet 8410e8b6b511403ea609b1353855937b
0503 14:12:12.863107 (+ 39us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.863242 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318095}
W20240503 14:12:12.864692 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 8410e8b6b511403ea609b1353855937b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8410e8b6b511403ea609b1353855937b already in progress: creating tablet
I20240503 14:12:12.865324 28653 ts_tablet_manager.cc:1888] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:12.865331 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 428) took 1245 ms. Trace:
I20240503 14:12:12.865576 28653 log.cc:1196] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/13c875ec8c5e49d1bb20a23dd7a21902
I20240503 14:12:12.865497 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dbb59d98834ab6bbac4abfa92681e3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:12.865461 28603 rpcz_store.cc:276] 0503 14:12:11.620151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.620232 (+ 81us) service_pool.cc:225] Handling call
0503 14:12:12.865095 (+1244863us) ts_tablet_manager.cc:1120] Deleting tablet 004d556f3ed44a7db1a52d205ecfd90b
0503 14:12:12.865107 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.865238 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244496}
I20240503 14:12:12.866226 28653 ts_tablet_manager.cc:1909] T 13c875ec8c5e49d1bb20a23dd7a21902 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:12.866642 28150 leader_election.cc:341] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:12.866770 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 004d556f3ed44a7db1a52d205ecfd90b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 004d556f3ed44a7db1a52d205ecfd90b already in progress: creating tablet
I20240503 14:12:12.867079 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dbb59d98834ab6bbac4abfa92681e3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:12.867897 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 11ac912a3bd64cdc9082596e09c72e13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11ac912a3bd64cdc9082596e09c72e13 already in progress: creating tablet
W20240503 14:12:12.868724 28150 leader_election.cc:341] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:12.868947 28150 leader_election.cc:302] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:12.869349 29281 raft_consensus.cc:2726] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:12.869869 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ff4d28d3c2ea472191ad21355248a2fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4d28d3c2ea472191ad21355248a2fc already in progress: creating tablet
W20240503 14:12:12.871004 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9ac2833a0bb045dcb2824468d936ffe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ac2833a0bb045dcb2824468d936ffe4 already in progress: creating tablet
W20240503 14:12:12.872211 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 54f08aac9a7147ebaffda4262a57a86a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f08aac9a7147ebaffda4262a57a86a already in progress: creating tablet
W20240503 14:12:12.873667 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0af0342e0e0b47499a9322b5b2cb3d03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0af0342e0e0b47499a9322b5b2cb3d03 already in progress: creating tablet
I20240503 14:12:12.874487 28603 tablet_replica.cc:330] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.874653 28600 ts_tablet_manager.cc:1398] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:12.875149 28603 raft_consensus.cc:2229] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.875380 28600 tablet_bootstrap.cc:492] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:12.875839 28603 raft_consensus.cc:2258] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.876901 28603 ts_tablet_manager.cc:1875] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.867357 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 399) took 2360 ms. Trace:
I20240503 14:12:12.880697 28653 rpcz_store.cc:276] 0503 14:12:10.506435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.506483 (+ 48us) service_pool.cc:225] Handling call
0503 14:12:12.800435 (+2293952us) ts_tablet_manager.cc:1120] Deleting tablet 13c875ec8c5e49d1bb20a23dd7a21902
0503 14:12:12.800446 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.855408 (+ 54962us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.865144 (+ 9736us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.867314 (+ 2170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2293773}
I20240503 14:12:12.881614 29295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:12.882552 28614 tablet_bootstrap.cc:492] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:12.882757 28614 ts_tablet_manager.cc:1367] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.001s
I20240503 14:12:12.883764 28600 tablet_bootstrap.cc:654] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.883714 28614 raft_consensus.cc:348] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.883987 28614 raft_consensus.cc:374] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.884318 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]) successfully deleted
I20240503 14:12:12.884853 28614 raft_consensus.cc:729] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.885254 28614 consensus_queue.cc:260] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.886523 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 401) took 2278 ms. Trace:
I20240503 14:12:12.887229 28614 ts_tablet_manager.cc:1398] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:12.888514 28614 tablet_bootstrap.cc:492] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:12.890384 29295 hybrid_clock.cc:648] HybridClock initialized: now 1714745532890187 us; error 103 us; skew 500 ppm
I20240503 14:12:12.891016 29295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:12.899582 29364 consensus_queue.cc:1035] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:12.904752 28653 rpcz_store.cc:276] 0503 14:12:10.607759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.607817 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:12.886227 (+2278410us) ts_tablet_manager.cc:1120] Deleting tablet 4e8411f474c247c2abfd0bd9682fb8a2
0503 14:12:12.886241 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.886455 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2278134}
I20240503 14:12:12.906090 28614 tablet_bootstrap.cc:654] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:12.908933 28604 tablet_bootstrap.cc:492] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:12.913182 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 402) took 2294 ms. Trace:
W20240503 14:12:12.913937 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 4e8411f474c247c2abfd0bd9682fb8a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e8411f474c247c2abfd0bd9682fb8a2 already in progress: creating tablet
I20240503 14:12:12.926465 28653 rpcz_store.cc:276] 0503 14:12:10.618294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.618357 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:12.912946 (+2294589us) ts_tablet_manager.cc:1120] Deleting tablet 646f83185d5241268d4126d10561e937
0503 14:12:12.912957 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.913120 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2294278}
W20240503 14:12:12.927384 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 646f83185d5241268d4126d10561e937 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 646f83185d5241268d4126d10561e937 already in progress: creating tablet
I20240503 14:12:12.927417 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 404) took 2224 ms. Trace:
I20240503 14:12:12.927628 28653 rpcz_store.cc:276] 0503 14:12:10.703003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.703466 (+ 463us) service_pool.cc:225] Handling call
0503 14:12:12.926892 (+2223426us) ts_tablet_manager.cc:1120] Deleting tablet 49445f0f1dd1451c9ba066e61273f2f1
0503 14:12:12.926905 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.927060 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2223109}
I20240503 14:12:12.928234 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 405) took 2183 ms. Trace:
W20240503 14:12:12.928301 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 49445f0f1dd1451c9ba066e61273f2f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49445f0f1dd1451c9ba066e61273f2f1 already in progress: creating tablet
I20240503 14:12:12.928625 28604 ts_tablet_manager.cc:1367] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.184s user 0.013s sys 0.015s
I20240503 14:12:12.930408 28604 raft_consensus.cc:348] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.930831 28604 raft_consensus.cc:374] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.931021 28604 raft_consensus.cc:729] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.931409 28604 consensus_queue.cc:260] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.933744 28604 ts_tablet_manager.cc:1398] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:12.937922 29420 tablet.cc:1722] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:12.946741 28603 ts_tablet_manager.cc:1888] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:12.954133 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6c90869b5034128864faaf05697c230 consensus state has no leader
I20240503 14:12:12.954705 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7667eac51a7246a595705267a0d758a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.955371 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7667eac51a7246a595705267a0d758a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.956760 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e3e1acbcbb7c4aeab3db92cc253b441d" } modify_external_catalogs: true
I20240503 14:12:12.957023 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e3e1acbcbb7c4aeab3db92cc253b441d" } modify_external_catalogs: true
I20240503 14:12:12.958117 28661 ts_tablet_manager.cc:1888] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:12.958324 28661 log.cc:1196] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e9609112657f427cbeeb3ef8f28cddf0
I20240503 14:12:12.960660 28603 log.cc:1196] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/f50ad403bd354559b2522a76af1fa722
I20240503 14:12:12.961778 28603 ts_tablet_manager.cc:1909] T f50ad403bd354559b2522a76af1fa722 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:12.963127 28214 raft_consensus.cc:1261] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:12.963997 28653 rpcz_store.cc:276] 0503 14:12:10.744734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.744801 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:12.928004 (+2183203us) ts_tablet_manager.cc:1120] Deleting tablet d6f5103b31ed45b480252dcf94b72f9c
0503 14:12:12.928016 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.928168 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2182926}
I20240503 14:12:12.964258 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet f50ad403bd354559b2522a76af1fa722 (table default.Table_5eec9210201643c3a011af892345d421 [id=f4c76f2fa96d41c491fb0ac30f49e43f]) successfully deleted
I20240503 14:12:12.964699 28603 tablet_replica.cc:330] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:12.964706 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 406) took 2219 ms. Trace:
W20240503 14:12:12.964908 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d6f5103b31ed45b480252dcf94b72f9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6f5103b31ed45b480252dcf94b72f9c already in progress: creating tablet
I20240503 14:12:12.965114 28603 raft_consensus.cc:2229] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:12.964962 28653 rpcz_store.cc:276] 0503 14:12:10.745684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.748149 (+ 2465us) service_pool.cc:225] Handling call
0503 14:12:12.964479 (+2216330us) ts_tablet_manager.cc:1120] Deleting tablet 6696a237c3dd428097e2ebdf24afd63f
0503 14:12:12.964491 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.964637 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216046}
W20240503 14:12:12.965286 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d7c79ffee3954b5c870f4a8aa6379517 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c79ffee3954b5c870f4a8aa6379517 already in progress: creating tablet
I20240503 14:12:12.965327 28603 raft_consensus.cc:2258] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:12.964843 28614 tablet_bootstrap.cc:492] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:12.965517 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 407) took 2216 ms. Trace:
I20240503 14:12:12.965677 28614 ts_tablet_manager.cc:1367] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.007s
I20240503 14:12:12.965682 28653 rpcz_store.cc:276] 0503 14:12:10.749115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:10.749175 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:12.965314 (+2216139us) ts_tablet_manager.cc:1120] Deleting tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd
0503 14:12:12.965324 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.965453 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2215900}
W20240503 14:12:12.965920 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6696a237c3dd428097e2ebdf24afd63f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6696a237c3dd428097e2ebdf24afd63f already in progress: creating tablet
I20240503 14:12:12.966064 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 420) took 1670 ms. Trace:
I20240503 14:12:12.966207 28653 rpcz_store.cc:276] 0503 14:12:11.295455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.296135 (+ 680us) service_pool.cc:225] Handling call
0503 14:12:12.965914 (+1669779us) ts_tablet_manager.cc:1120] Deleting tablet 8b1728b6f0e64661aaa8184ce51366d2
0503 14:12:12.965924 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.965999 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1669374}
W20240503 14:12:12.966498 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd already in progress: creating tablet
I20240503 14:12:12.966581 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 421) took 1670 ms. Trace:
I20240503 14:12:12.966727 28653 rpcz_store.cc:276] 0503 14:12:11.296021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.356716 (+ 60695us) service_pool.cc:225] Handling call
0503 14:12:12.966417 (+1609701us) ts_tablet_manager.cc:1120] Deleting tablet 0a50b3e199314bfc8465fdb7b2425d07
0503 14:12:12.966426 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.966504 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1609266}
I20240503 14:12:12.967403 29421 tablet.cc:1722] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:12.968626 28603 ts_tablet_manager.cc:1875] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:12.966689 28614 raft_consensus.cc:348] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.971064 28614 raft_consensus.cc:374] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.972301 29426 tablet.cc:1722] T 23186a7a0dfb432e813c8a05e97e46dd P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:12.976588 28604 tablet_bootstrap.cc:492] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:12.977521 29317 consensus_queue.cc:1035] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:12:12.979396 29428 tablet.cc:1722] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:12.980602 28614 raft_consensus.cc:729] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.981034 28614 consensus_queue.cc:260] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.981735 29295 webserver.cc:460] Webserver started at http://127.27.50.253:43185/ using document root <none> and password file <none>
I20240503 14:12:12.981947 28600 tablet_bootstrap.cc:492] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:12.982223 28600 ts_tablet_manager.cc:1367] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.107s user 0.001s sys 0.004s
W20240503 14:12:12.983295 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2cad43f3c6094253ba87d0ed3fc29fd9 consensus state has no leader
I20240503 14:12:12.983558 28600 raft_consensus.cc:348] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.983831 28600 raft_consensus.cc:374] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:12.983968 28600 raft_consensus.cc:729] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:12.983891 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0ac41173b58543dcaa1508ac89ee3a4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:12.984586 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ac41173b58543dcaa1508ac89ee3a4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.986011 28031 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a1ed178c064946af8a771daa00684ff2
I20240503 14:12:12.986269 28600 consensus_queue.cc:260] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:12.986510 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a2bb32853fa94122b1e4188e3fc4f65e" } modify_external_catalogs: true
I20240503 14:12:12.986750 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a2bb32853fa94122b1e4188e3fc4f65e" } modify_external_catalogs: true
I20240503 14:12:12.987478 29295 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:12.987731 29295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:12.988533 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9f63e6f47cb4390b4fff24fd5d4f83e" } new_table_name: "default.Table_ab495b10ad82425ba5c7123a6e1cf064" modify_external_catalogs: true
I20240503 14:12:12.989771 28182 tablet_service.cc:1467] Processing CreateTablet for tablet f0287bc315644f0aaa976a56174576d9 (DEFAULT_TABLE table=default.Table_a28d952dbd9141a9a113c42bcbc27a14 [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.990267 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0287bc315644f0aaa976a56174576d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.990680 28327 tablet_service.cc:1467] Processing CreateTablet for tablet cc939b81d7cc472aa73b51d9dc397013 (DEFAULT_TABLE table=default.Table_71db0c0f4c9f475fb810a40fff00d5a2 [id=a77063b0895c4ad99855c1ded7332378]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.991429 28318 tablet_service.cc:1467] Processing CreateTablet for tablet e8f9f10348014293a421ed11a50f89e7 (DEFAULT_TABLE table=default.Table_0a7f788e080745b2bf2fe782dfae1bbd [id=bd1dc8015f814e798f87b4dc4c0ee715]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.991681 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 7945118bb17b431dac8670007d2d0a18 (table default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]). Expected version 1 got 0
I20240503 14:12:12.992049 28336 tablet_service.cc:1467] Processing CreateTablet for tablet cf221f17ddfd461f83f7f916d7833011 (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.992406 28180 tablet_service.cc:1467] Processing CreateTablet for tablet cc939b81d7cc472aa73b51d9dc397013 (DEFAULT_TABLE table=default.Table_71db0c0f4c9f475fb810a40fff00d5a2 [id=a77063b0895c4ad99855c1ded7332378]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.992620 28325 tablet_service.cc:1467] Processing CreateTablet for tablet dc31b2e907b04ea19a6f9f7fd8f57843 (DEFAULT_TABLE table=default.Table_96e46f964a63484bb1f66f90e65c72db [id=b9f35d30fe7d417db20cbde7e386d870]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.992707 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9360be1891a744a48835f3976afd1059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:12.992880 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc939b81d7cc472aa73b51d9dc397013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.993167 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 (DEFAULT_TABLE table=default.Table_8e709ec40c1b4306944f2be34700e617 [id=ee1aab56b13743209d7815332cf50e69]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:12.993252 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9360be1891a744a48835f3976afd1059 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:12.994628 28195 tablet_service.cc:1467] Processing CreateTablet for tablet e8f9f10348014293a421ed11a50f89e7 (DEFAULT_TABLE table=default.Table_0a7f788e080745b2bf2fe782dfae1bbd [id=bd1dc8015f814e798f87b4dc4c0ee715]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.995172 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f9f10348014293a421ed11a50f89e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.996491 28193 tablet_service.cc:1467] Processing CreateTablet for tablet cf221f17ddfd461f83f7f916d7833011 (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.997004 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf221f17ddfd461f83f7f916d7833011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:12.997713 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 491f882bc58b47b8868ade2e2a1f6cf2 (DEFAULT_TABLE table=default.Table_05db2dcd653343c0ac8992415e36c7d6 [id=7a4e873e0ff04101982fef78f2a8b31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:12.999469 28329 tablet_service.cc:1467] Processing CreateTablet for tablet f0287bc315644f0aaa976a56174576d9 (DEFAULT_TABLE table=default.Table_a28d952dbd9141a9a113c42bcbc27a14 [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.001490 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 (DEFAULT_TABLE table=default.Table_2e27287910d04e1b9c08ae3eca4739b0 [id=29b2ba9909c34f528bdbb8ef06268373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.003985 28661 ts_tablet_manager.cc:1909] T e9609112657f427cbeeb3ef8f28cddf0 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:13.005446 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 398) took 2230 ms. Trace:
I20240503 14:12:13.005677 28661 rpcz_store.cc:276] 0503 14:12:10.774871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.301179 (+526308us) service_pool.cc:225] Handling call
0503 14:12:12.720640 (+1419461us) ts_tablet_manager.cc:1120] Deleting tablet e9609112657f427cbeeb3ef8f28cddf0
0503 14:12:12.720651 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:12.909197 (+188546us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:12.957926 (+ 48729us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.005384 (+ 47458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408773}
I20240503 14:12:13.006311 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 406) took 2143 ms. Trace:
I20240503 14:12:13.006526 28661 rpcz_store.cc:276] 0503 14:12:10.863172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.311891 (+448719us) service_pool.cc:225] Handling call
0503 14:12:13.006088 (+1694197us) ts_tablet_manager.cc:1120] Deleting tablet 6696a237c3dd428097e2ebdf24afd63f
0503 14:12:13.006098 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.006223 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693880}
I20240503 14:12:13.007351 28458 tablet_service.cc:1467] Processing CreateTablet for tablet f0287bc315644f0aaa976a56174576d9 (DEFAULT_TABLE table=default.Table_a28d952dbd9141a9a113c42bcbc27a14 [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.007899 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0287bc315644f0aaa976a56174576d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.008308 28461 tablet_service.cc:1467] Processing CreateTablet for tablet dc31b2e907b04ea19a6f9f7fd8f57843 (DEFAULT_TABLE table=default.Table_96e46f964a63484bb1f66f90e65c72db [id=b9f35d30fe7d417db20cbde7e386d870]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.009021 28465 tablet_service.cc:1467] Processing CreateTablet for tablet cc939b81d7cc472aa73b51d9dc397013 (DEFAULT_TABLE table=default.Table_71db0c0f4c9f475fb810a40fff00d5a2 [id=a77063b0895c4ad99855c1ded7332378]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.009616 28469 tablet_service.cc:1467] Processing CreateTablet for tablet e8f9f10348014293a421ed11a50f89e7 (DEFAULT_TABLE table=default.Table_0a7f788e080745b2bf2fe782dfae1bbd [id=bd1dc8015f814e798f87b4dc4c0ee715]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.009618 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e9609112657f427cbeeb3ef8f28cddf0 (table default.Table_633e72d5fd384b689e09a7eef5a8d1d9 [id=500a3febdac042138ea31f6f3c8d02b5]) successfully deleted
I20240503 14:12:13.010210 28467 tablet_service.cc:1467] Processing CreateTablet for tablet cf221f17ddfd461f83f7f916d7833011 (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.010752 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 (DEFAULT_TABLE table=default.Table_8e709ec40c1b4306944f2be34700e617 [id=ee1aab56b13743209d7815332cf50e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.009637 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]). Expected version 1 got 0
I20240503 14:12:13.011312 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 491f882bc58b47b8868ade2e2a1f6cf2 (DEFAULT_TABLE table=default.Table_05db2dcd653343c0ac8992415e36c7d6 [id=7a4e873e0ff04101982fef78f2a8b31d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:13.011483 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6696a237c3dd428097e2ebdf24afd63f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6696a237c3dd428097e2ebdf24afd63f already in progress: creating tablet
I20240503 14:12:13.011548 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 7945118bb17b431dac8670007d2d0a18 (table default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]). Expected version 1 got 0
I20240503 14:12:13.011878 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 (DEFAULT_TABLE table=default.Table_2e27287910d04e1b9c08ae3eca4739b0 [id=29b2ba9909c34f528bdbb8ef06268373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.014806 28188 tablet_service.cc:1467] Processing CreateTablet for tablet dc31b2e907b04ea19a6f9f7fd8f57843 (DEFAULT_TABLE table=default.Table_96e46f964a63484bb1f66f90e65c72db [id=b9f35d30fe7d417db20cbde7e386d870]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.014806 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 491f882bc58b47b8868ade2e2a1f6cf2 (DEFAULT_TABLE table=default.Table_05db2dcd653343c0ac8992415e36c7d6 [id=7a4e873e0ff04101982fef78f2a8b31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.015343 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc31b2e907b04ea19a6f9f7fd8f57843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.015970 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491f882bc58b47b8868ade2e2a1f6cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.016126 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 (DEFAULT_TABLE table=default.Table_8e709ec40c1b4306944f2be34700e617 [id=ee1aab56b13743209d7815332cf50e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.016548 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491f882bc58b47b8868ade2e2a1f6cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.017751 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.020262 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf221f17ddfd461f83f7f916d7833011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.023275 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 7945118bb17b431dac8670007d2d0a18 (table default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]). Expected version 1 got 0
I20240503 14:12:13.024325 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f6fb009bdd4f70ba90e7b5f972cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.028577 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f9f10348014293a421ed11a50f89e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.032486 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc939b81d7cc472aa73b51d9dc397013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.033437 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 (DEFAULT_TABLE table=default.Table_2e27287910d04e1b9c08ae3eca4739b0 [id=29b2ba9909c34f528bdbb8ef06268373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.036376 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.040736 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc31b2e907b04ea19a6f9f7fd8f57843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.041424 29295 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.006s sys 0.000s
I20240503 14:12:13.042796 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f6fb009bdd4f70ba90e7b5f972cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.057215 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 86804987bdf340dc83ddd58ba2bc8a96 (DEFAULT_TABLE table=default.Table_42ee02ce6afe4ddb8480ec52a7d8993f [id=9c06c541eab341558ee262552c52b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.058640 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 86804987bdf340dc83ddd58ba2bc8a96 (DEFAULT_TABLE table=default.Table_42ee02ce6afe4ddb8480ec52a7d8993f [id=9c06c541eab341558ee262552c52b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.059105 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 320c526e845e4dc78da345bcb095918f (DEFAULT_TABLE table=default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.059211 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86804987bdf340dc83ddd58ba2bc8a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.059453 29435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.059880 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 86804987bdf340dc83ddd58ba2bc8a96 (DEFAULT_TABLE table=default.Table_42ee02ce6afe4ddb8480ec52a7d8993f [id=9c06c541eab341558ee262552c52b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.060544 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86804987bdf340dc83ddd58ba2bc8a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.062356 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 320c526e845e4dc78da345bcb095918f (DEFAULT_TABLE table=default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.062820 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320c526e845e4dc78da345bcb095918f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.063127 29295 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.006s sys 0.002s
I20240503 14:12:13.063338 29295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
uuid: "a10ddca38c26462eb221d6026e88c6cc"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-8wrt"
I20240503 14:12:13.064750 29295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:13.065002 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 320c526e845e4dc78da345bcb095918f (DEFAULT_TABLE table=default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.065588 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 8097ba5606cf42d68448833de36d85c7 (DEFAULT_TABLE table=default.Table_b86f741291e64aef9908c5e036e4cce8 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.065811 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 8097ba5606cf42d68448833de36d85c7 (DEFAULT_TABLE table=default.Table_b86f741291e64aef9908c5e036e4cce8 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.066112 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8097ba5606cf42d68448833de36d85c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.066826 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc939b81d7cc472aa73b51d9dc397013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.067414 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0287bc315644f0aaa976a56174576d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.068434 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320c526e845e4dc78da345bcb095918f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.070250 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 8097ba5606cf42d68448833de36d85c7 (DEFAULT_TABLE table=default.Table_b86f741291e64aef9908c5e036e4cce8 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.071676 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8097ba5606cf42d68448833de36d85c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.072322 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8097ba5606cf42d68448833de36d85c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.076053 28187 tablet_service.cc:1467] Processing CreateTablet for tablet c9c0540960ac45aabeaf13f888caf238 (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.076673 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c0540960ac45aabeaf13f888caf238. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:13.076670 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 (table default.Table_a9e8bd96c6824e9c8c64036f0848c876 [id=9e438ac1c3254c7aae6f81c707f433d7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2cad43f3c6094253ba87d0ed3fc29fd9 consensus state has no leader
I20240503 14:12:13.082500 28464 tablet_service.cc:1467] Processing CreateTablet for tablet d04ce185de404299b4f66afd906ece80 (DEFAULT_TABLE table=default.Table_a6709087fc714b21979fd6349c864785 [id=487a10eede214c358884b0f9ed3b83dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.083556 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04ce185de404299b4f66afd906ece80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.083493 28181 tablet_service.cc:1467] Processing CreateTablet for tablet d04ce185de404299b4f66afd906ece80 (DEFAULT_TABLE table=default.Table_a6709087fc714b21979fd6349c864785 [id=487a10eede214c358884b0f9ed3b83dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.084015 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04ce185de404299b4f66afd906ece80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.087963 28330 tablet_service.cc:1467] Processing CreateTablet for tablet c9c0540960ac45aabeaf13f888caf238 (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.088410 28324 tablet_service.cc:1467] Processing CreateTablet for tablet d04ce185de404299b4f66afd906ece80 (DEFAULT_TABLE table=default.Table_a6709087fc714b21979fd6349c864785 [id=487a10eede214c358884b0f9ed3b83dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.088708 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320c526e845e4dc78da345bcb095918f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.089273 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.089515 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc31b2e907b04ea19a6f9f7fd8f57843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.092358 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04ce185de404299b4f66afd906ece80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.092701 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86804987bdf340dc83ddd58ba2bc8a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.096220 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf221f17ddfd461f83f7f916d7833011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.096534 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c0540960ac45aabeaf13f888caf238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.110265 28462 tablet_service.cc:1467] Processing CreateTablet for tablet c9c0540960ac45aabeaf13f888caf238 (DEFAULT_TABLE table=default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb [id=f58e9b32e96948cfab2760b64eb18b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.111491 28600 ts_tablet_manager.cc:1398] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.129s user 0.000s sys 0.002s
I20240503 14:12:13.112125 28600 tablet_bootstrap.cc:492] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:13.121549 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_446afb3f188b43f1aea50e0337a2c793" } new_table_name: "default.Table_a6b8bd2eddaf402f992d809054a1bae3" modify_external_catalogs: true
I20240503 14:12:13.126597 28603 ts_tablet_manager.cc:1888] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.126832 28603 log.cc:1196] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/605531ed29214b1ca27011c5f5e96b11
I20240503 14:12:13.130841 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491f882bc58b47b8868ade2e2a1f6cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.131417 28603 ts_tablet_manager.cc:1909] T 605531ed29214b1ca27011c5f5e96b11 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:13.132138 29317 raft_consensus.cc:482] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.132333 29317 raft_consensus.cc:504] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.133111 29317 leader_election.cc:288] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.133818 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77f6fb009bdd4f70ba90e7b5f972cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.133869 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c0540960ac45aabeaf13f888caf238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.133780 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 70f0e07d32084999a3ff42f69bb1ca2f (DEFAULT_TABLE table=default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.134629 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 2a90167ae51647688dfc502b00c5c9c6 (DEFAULT_TABLE table=default.Table_21ea716e646a43279621abff726dedd1 [id=6d13d806de0247b5b10b39405fa05ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.135113 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a90167ae51647688dfc502b00c5c9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.139431 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 70f0e07d32084999a3ff42f69bb1ca2f (DEFAULT_TABLE table=default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.140137 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f0e07d32084999a3ff42f69bb1ca2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.143013 28614 ts_tablet_manager.cc:1398] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.163s user 0.002s sys 0.001s
I20240503 14:12:13.143810 29420 tablet.cc:1722] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:13.144677 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f9f10348014293a421ed11a50f89e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.150329 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 605531ed29214b1ca27011c5f5e96b11 (table default.Table_2907862ac0af4c3692460a0071e8bce3 [id=651246fc28e144af9b8a0bbe9926aa22]) successfully deleted
W20240503 14:12:13.152396 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba already in progress: creating tablet
I20240503 14:12:13.158902 28604 tablet_bootstrap.cc:654] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:13.164664 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0a50b3e199314bfc8465fdb7b2425d07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a50b3e199314bfc8465fdb7b2425d07 already in progress: creating tablet
I20240503 14:12:13.168869 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 407) took 2303 ms. Trace:
I20240503 14:12:13.169123 28661 rpcz_store.cc:276] 0503 14:12:10.865782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.312234 (+446452us) service_pool.cc:225] Handling call
0503 14:12:13.006915 (+1694681us) ts_tablet_manager.cc:1120] Deleting tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd
0503 14:12:13.006923 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.168765 (+161842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1694486}
I20240503 14:12:13.170001 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 399) took 2394 ms. Trace:
I20240503 14:12:13.170236 28661 rpcz_store.cc:276] 0503 14:12:10.774957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.301791 (+526834us) service_pool.cc:225] Handling call
0503 14:12:13.169701 (+1867910us) ts_tablet_manager.cc:1120] Deleting tablet 00b52ea336524beea2f255c8720f7472
0503 14:12:13.169714 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.169908 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1856753}
I20240503 14:12:13.170850 28661 tablet_replica.cc:330] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:13.170907 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f0e07d32084999a3ff42f69bb1ca2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.171155 28661 raft_consensus.cc:2229] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.171433 28661 raft_consensus.cc:2258] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.172268 28661 ts_tablet_manager.cc:1875] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:13.172753 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd already in progress: creating tablet
W20240503 14:12:13.173311 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 00b52ea336524beea2f255c8720f7472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00b52ea336524beea2f255c8720f7472 already in progress: creating tablet
W20240503 14:12:13.173575 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet daf5d153ab5142c0902cc6e96629f412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf5d153ab5142c0902cc6e96629f412 already in progress: creating tablet
I20240503 14:12:13.173722 28653 tablet_replica.cc:330] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:13.173862 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 already in progress: creating tablet
W20240503 14:12:13.174119 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c816f3b05cfb4496984f6cb28d310cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c816f3b05cfb4496984f6cb28d310cc4 already in progress: creating tablet
I20240503 14:12:13.175621 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 70f0e07d32084999a3ff42f69bb1ca2f (DEFAULT_TABLE table=default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.178122 29427 tablet.cc:1722] T 23186a7a0dfb432e813c8a05e97e46dd P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:13.185065 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_afbdf248811d4722b5fa2c30b6f9780f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.185781 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afbdf248811d4722b5fa2c30b6f9780f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.187199 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 320c526e845e4dc78da345bcb095918f
W20240503 14:12:13.188410 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77c77a11ef384709aef5b62273380573 consensus state has no leader
I20240503 14:12:13.189191 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 4084bd4fbf604042bb33ff03c3f8cc19
I20240503 14:12:13.189617 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 320c526e845e4dc78da345bcb095918f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.190601 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_8b6756d9ceab442784c7aa1436e25032 [id=68a4a475afc24b95b5388e43deb4632b] alter complete (version 2)
I20240503 14:12:13.190866 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.198961 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7945118bb17b431dac8670007d2d0a18" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:13.199313 28217 raft_consensus.cc:2454] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:13.199865 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ca6627f0d9a544bb8176bea6c1af5ab4
I20240503 14:12:13.201517 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_507e44ddf53d4f36bf1c7179093063ff" } new_table_name: "default.Table_92abfaed91ab4683a6b2e501ce673e58" modify_external_catalogs: true
I20240503 14:12:13.203641 28614 tablet_bootstrap.cc:492] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:13.205904 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.215071 28427 leader_election.cc:302] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:13.216068 29317 raft_consensus.cc:2781] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.216418 29317 raft_consensus.cc:482] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.216528 29317 raft_consensus.cc:3037] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.219913 29317 raft_consensus.cc:504] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.221155 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 2a90167ae51647688dfc502b00c5c9c6 (DEFAULT_TABLE table=default.Table_21ea716e646a43279621abff726dedd1 [id=6d13d806de0247b5b10b39405fa05ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.221488 29317 leader_election.cc:288] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.222183 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8" } modify_external_catalogs: true
I20240503 14:12:13.222471 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8" } modify_external_catalogs: true
I20240503 14:12:13.223866 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet a6c17cfce7e644668817bf3483a098d4 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:13.224943 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 320c526e845e4dc78da345bcb095918f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.225127 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df
I20240503 14:12:13.225109 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7945118bb17b431dac8670007d2d0a18" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:13.225688 28030 catalog_manager.cc:6554] ReplaceTablet: tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df deleted and replaced by tablet ae909ab6263a401490952e3f68533741
I20240503 14:12:13.226018 29406 raft_consensus.cc:482] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.226186 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9150e78ff2d94d1dba8f6d1fd1727db2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.226824 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9150e78ff2d94d1dba8f6d1fd1727db2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.227658 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df with delete_type TABLET_DATA_DELETED (replaced by tablet ae909ab6263a401490952e3f68533741) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.228181 28356 raft_consensus.cc:2454] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:13.229696 28603 tablet_replica.cc:330] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:13.230513 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_52c51767da7d4d86b088b1bb16ce9623"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.231213 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52c51767da7d4d86b088b1bb16ce9623 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.232453 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_df4c84dc5f5f425b9f6fe90807b5f1fb" } new_table_name: "default.Table_8f505fa519514278ae7489b9ff7bbaf2" modify_external_catalogs: true
W20240503 14:12:13.233667 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 007c84fd9d9247728b0babd48689aba8 consensus state has no leader
I20240503 14:12:13.233675 28603 raft_consensus.cc:2229] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.234231 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_45877d24af954c5ebdd54673110911e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.234823 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45877d24af954c5ebdd54673110911e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.231864 29406 raft_consensus.cc:504] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.236408 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6b19c8158c64491dad1fdcddefb6b2b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.237041 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b19c8158c64491dad1fdcddefb6b2b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.237630 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.238313 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36] alter complete (version 1)
I20240503 14:12:13.238600 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.238873 29406 leader_election.cc:288] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.239040 28603 raft_consensus.cc:2258] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.239343 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c297c309fedc4148849b57c91f68d1b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:13.239483 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fb79fcf30d485798c64c815e372dc1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.239714 28603 ts_tablet_manager.cc:1875] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:13.239933 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c297c309fedc4148849b57c91f68d1b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.241415 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265dc5f2630e453ea71abddb1a9753f3" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:13.241722 28491 raft_consensus.cc:2454] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:13.254133 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a90167ae51647688dfc502b00c5c9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.257975 29295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:13.258496 28661 ts_tablet_manager.cc:1888] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:13.259246 28661 log.cc:1196] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/13c875ec8c5e49d1bb20a23dd7a21902
I20240503 14:12:13.260656 28661 ts_tablet_manager.cc:1909] T 13c875ec8c5e49d1bb20a23dd7a21902 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:13.261875 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 400) took 2486 ms. Trace:
I20240503 14:12:13.263828 28661 rpcz_store.cc:276] 0503 14:12:10.775041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.302649 (+527608us) service_pool.cc:225] Handling call
0503 14:12:13.170728 (+1868079us) ts_tablet_manager.cc:1120] Deleting tablet 13c875ec8c5e49d1bb20a23dd7a21902
0503 14:12:13.170739 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.243217 (+ 72478us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.258277 (+ 15060us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.261827 (+ 3550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1856869}
I20240503 14:12:13.263926 28290 leader_election.cc:302] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:13.266204 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcd405fd84d44076b1792b7fcbe00e84 consensus state has no leader
I20240503 14:12:13.266696 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 13c875ec8c5e49d1bb20a23dd7a21902 (table default.Table_ddcf99aec7384d15b0149124ff7a6646 [id=8961b0b7af2a4e999ee705db8d8e2bf0]) successfully deleted
I20240503 14:12:13.267398 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 401) took 2492 ms. Trace:
I20240503 14:12:13.268177 28661 rpcz_store.cc:276] 0503 14:12:10.775122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.303306 (+528184us) service_pool.cc:225] Handling call
0503 14:12:13.267170 (+1963864us) ts_tablet_manager.cc:1120] Deleting tablet 4e8411f474c247c2abfd0bd9682fb8a2
0503 14:12:13.267182 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.267335 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1952070}
I20240503 14:12:13.267611 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df with delete_type TABLET_DATA_DELETED (replaced by tablet ae909ab6263a401490952e3f68533741) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.269318 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f0e07d32084999a3ff42f69bb1ca2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:13.269598 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 4e8411f474c247c2abfd0bd9682fb8a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e8411f474c247c2abfd0bd9682fb8a2 already in progress: creating tablet
I20240503 14:12:13.270387 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 402) took 2495 ms. Trace:
I20240503 14:12:13.271553 28661 rpcz_store.cc:276] 0503 14:12:10.775213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.304151 (+528938us) service_pool.cc:225] Handling call
0503 14:12:13.270137 (+1965986us) ts_tablet_manager.cc:1120] Deleting tablet 646f83185d5241268d4126d10561e937
0503 14:12:13.270149 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.270325 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1954637}
W20240503 14:12:13.274200 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 646f83185d5241268d4126d10561e937 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 646f83185d5241268d4126d10561e937 already in progress: creating tablet
I20240503 14:12:13.272912 28604 tablet_bootstrap.cc:492] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:13.274741 28604 ts_tablet_manager.cc:1367] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.340s user 0.006s sys 0.010s
I20240503 14:12:13.275722 28604 raft_consensus.cc:348] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.276158 28604 raft_consensus.cc:374] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.276325 28604 raft_consensus.cc:729] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.276811 28604 consensus_queue.cc:260] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.278362 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]). Expected version 2 got 0
I20240503 14:12:13.278936 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 651478ef632144079f93f3b50bff4611
I20240503 14:12:13.279871 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9ecfb667ad96405c894e48eda793fada" } new_table_name: "default.Table_04ae17bd98c04c4b993d808882cdc3ec" modify_external_catalogs: true
I20240503 14:12:13.282658 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 651478ef632144079f93f3b50bff4611 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.283210 28653 raft_consensus.cc:2229] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.283514 28653 raft_consensus.cc:2258] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.283879 28604 ts_tablet_manager.cc:1398] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:12:13.285070 29406 raft_consensus.cc:2781] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.285290 28614 tablet_bootstrap.cc:654] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.286698 29440 tablet.cc:1722] T 23186a7a0dfb432e813c8a05e97e46dd P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:13.287382 29432 tablet.cc:1722] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:13.288121 28290 leader_election.cc:341] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 265dc5f2630e453ea71abddb1a9753f3
I20240503 14:12:13.290071 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.290421 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.290756 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 651478ef632144079f93f3b50bff4611 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.290952 28604 tablet_bootstrap.cc:492] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:13.292150 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 2a90167ae51647688dfc502b00c5c9c6 (DEFAULT_TABLE table=default.Table_21ea716e646a43279621abff726dedd1 [id=6d13d806de0247b5b10b39405fa05ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.292618 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a90167ae51647688dfc502b00c5c9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.292781 29406 raft_consensus.cc:482] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.292937 29406 raft_consensus.cc:3037] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.293406 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 320c526e845e4dc78da345bcb095918f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.293627 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df with delete_type TABLET_DATA_DELETED (replaced by tablet ae909ab6263a401490952e3f68533741) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.293903 28653 ts_tablet_manager.cc:1875] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.294613 28604 tablet_bootstrap.cc:654] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.298195 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 651478ef632144079f93f3b50bff4611 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.291075 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 404) took 2499 ms. Trace:
I20240503 14:12:13.299904 28661 rpcz_store.cc:276] 0503 14:12:10.791009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.305650 (+514641us) service_pool.cc:225] Handling call
0503 14:12:13.284145 (+1978495us) ts_tablet_manager.cc:1120] Deleting tablet 49445f0f1dd1451c9ba066e61273f2f1
0503 14:12:13.284153 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.290587 (+ 6434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968477}
W20240503 14:12:13.303551 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 49445f0f1dd1451c9ba066e61273f2f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49445f0f1dd1451c9ba066e61273f2f1 already in progress: creating tablet
I20240503 14:12:13.304183 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 405) took 2442 ms. Trace:
I20240503 14:12:13.304390 28661 rpcz_store.cc:276] 0503 14:12:10.861673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.306322 (+444649us) service_pool.cc:225] Handling call
0503 14:12:13.303956 (+1997634us) ts_tablet_manager.cc:1120] Deleting tablet d6f5103b31ed45b480252dcf94b72f9c
0503 14:12:13.303969 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.304126 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1988073}
I20240503 14:12:13.303433 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7945118bb17b431dac8670007d2d0a18" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:13.305132 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d6f5103b31ed45b480252dcf94b72f9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6f5103b31ed45b480252dcf94b72f9c already in progress: creating tablet
I20240503 14:12:13.305631 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 420) took 1802 ms. Trace:
I20240503 14:12:13.305835 28661 rpcz_store.cc:276] 0503 14:12:11.502930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.503089 (+ 159us) service_pool.cc:225] Handling call
0503 14:12:13.305430 (+1802341us) ts_tablet_manager.cc:1120] Deleting tablet a7295c2d3bd34e20a69986aadc9c7017
0503 14:12:13.305443 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.305576 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1789665}
W20240503 14:12:13.306488 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a7295c2d3bd34e20a69986aadc9c7017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7295c2d3bd34e20a69986aadc9c7017 already in progress: creating tablet
I20240503 14:12:13.306777 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7945118bb17b431dac8670007d2d0a18" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:13.306986 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 422) took 1788 ms. Trace:
I20240503 14:12:13.307132 28661 rpcz_store.cc:276] 0503 14:12:11.518527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.520327 (+ 1800us) service_pool.cc:225] Handling call
0503 14:12:13.306805 (+1786478us) ts_tablet_manager.cc:1120] Deleting tablet 0a50b3e199314bfc8465fdb7b2425d07
0503 14:12:13.306815 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.306931 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1783037}
W20240503 14:12:13.307624 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0a50b3e199314bfc8465fdb7b2425d07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a50b3e199314bfc8465fdb7b2425d07 already in progress: creating tablet
I20240503 14:12:13.308164 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 424) took 1789 ms. Trace:
I20240503 14:12:13.308353 28661 rpcz_store.cc:276] 0503 14:12:11.518779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.523804 (+ 5025us) service_pool.cc:225] Handling call
0503 14:12:13.307963 (+1784159us) ts_tablet_manager.cc:1120] Deleting tablet dbcc6cef7b7d464a9fbbae61caf0437d
0503 14:12:13.307972 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.308111 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1781011}
W20240503 14:12:13.308856 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet dbcc6cef7b7d464a9fbbae61caf0437d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbcc6cef7b7d464a9fbbae61caf0437d already in progress: creating tablet
I20240503 14:12:13.309388 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 421) took 1799 ms. Trace:
I20240503 14:12:13.309788 28661 rpcz_store.cc:276] 0503 14:12:11.510098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.534922 (+ 24824us) service_pool.cc:225] Handling call
0503 14:12:13.309201 (+1774279us) ts_tablet_manager.cc:1120] Deleting tablet 8b1728b6f0e64661aaa8184ce51366d2
0503 14:12:13.309212 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.309334 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773937}
I20240503 14:12:13.310633 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 425) took 1554 ms. Trace:
I20240503 14:12:13.310048 28352 raft_consensus.cc:3037] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.312194 28661 rpcz_store.cc:276] 0503 14:12:11.755760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.755873 (+ 113us) service_pool.cc:225] Handling call
0503 14:12:13.310463 (+1554590us) ts_tablet_manager.cc:1120] Deleting tablet 8410e8b6b511403ea609b1353855937b
0503 14:12:13.310472 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.310583 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1554138}
W20240503 14:12:13.312880 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 8410e8b6b511403ea609b1353855937b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8410e8b6b511403ea609b1353855937b already in progress: creating tablet
I20240503 14:12:13.313354 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 428) took 1547 ms. Trace:
I20240503 14:12:13.313537 28661 rpcz_store.cc:276] 0503 14:12:11.766271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.770467 (+ 4196us) service_pool.cc:225] Handling call
0503 14:12:13.313186 (+1542719us) ts_tablet_manager.cc:1120] Deleting tablet 004d556f3ed44a7db1a52d205ecfd90b
0503 14:12:13.313195 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.313301 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542314}
W20240503 14:12:13.314103 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 004d556f3ed44a7db1a52d205ecfd90b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 004d556f3ed44a7db1a52d205ecfd90b already in progress: creating tablet
I20240503 14:12:13.314599 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 434) took 1306 ms. Trace:
I20240503 14:12:13.315177 28214 raft_consensus.cc:3037] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.315436 28661 rpcz_store.cc:276] 0503 14:12:12.009299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.010234 (+ 935us) service_pool.cc:225] Handling call
0503 14:12:13.314420 (+1304186us) ts_tablet_manager.cc:1120] Deleting tablet 11ac912a3bd64cdc9082596e09c72e13
0503 14:12:13.314429 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.314542 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1303903}
W20240503 14:12:13.316510 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 11ac912a3bd64cdc9082596e09c72e13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11ac912a3bd64cdc9082596e09c72e13 already in progress: creating tablet
I20240503 14:12:13.317086 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 435) took 1260 ms. Trace:
I20240503 14:12:13.320837 28661 rpcz_store.cc:276] 0503 14:12:12.056497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.056572 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:13.316891 (+1260319us) ts_tablet_manager.cc:1120] Deleting tablet ff4d28d3c2ea472191ad21355248a2fc
0503 14:12:13.316902 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.317035 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1259949}
I20240503 14:12:13.321610 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 436) took 1264 ms. Trace:
I20240503 14:12:13.322027 28661 rpcz_store.cc:276] 0503 14:12:12.056688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.056988 (+ 300us) service_pool.cc:225] Handling call
0503 14:12:13.321408 (+1264420us) ts_tablet_manager.cc:1120] Deleting tablet 9ac2833a0bb045dcb2824468d936ffe4
0503 14:12:13.321420 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.321547 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264164}
I20240503 14:12:13.324406 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 450) took 1026 ms. Trace:
I20240503 14:12:13.324929 28661 rpcz_store.cc:276] 0503 14:12:12.297581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.443302 (+145721us) service_pool.cc:225] Handling call
0503 14:12:13.324248 (+880946us) ts_tablet_manager.cc:1120] Deleting tablet 0af0342e0e0b47499a9322b5b2cb3d03
0503 14:12:13.324258 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.324354 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":880546}
W20240503 14:12:13.326362 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ff4d28d3c2ea472191ad21355248a2fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4d28d3c2ea472191ad21355248a2fc already in progress: creating tablet
W20240503 14:12:13.326748 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9ac2833a0bb045dcb2824468d936ffe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ac2833a0bb045dcb2824468d936ffe4 already in progress: creating tablet
W20240503 14:12:13.326977 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet daf5d153ab5142c0902cc6e96629f412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf5d153ab5142c0902cc6e96629f412 already in progress: creating tablet
W20240503 14:12:13.327229 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 already in progress: creating tablet
W20240503 14:12:13.327461 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0af0342e0e0b47499a9322b5b2cb3d03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0af0342e0e0b47499a9322b5b2cb3d03 already in progress: creating tablet
W20240503 14:12:13.327733 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba already in progress: creating tablet
I20240503 14:12:13.327929 28604 tablet_bootstrap.cc:492] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:13.328195 28604 ts_tablet_manager.cc:1367] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.003s
I20240503 14:12:13.329263 28604 raft_consensus.cc:348] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.329602 28604 raft_consensus.cc:374] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.329874 28604 raft_consensus.cc:729] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.330204 28604 consensus_queue.cc:260] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.335734 28604 ts_tablet_manager.cc:1398] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:13.341411 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 449) took 1030 ms. Trace:
I20240503 14:12:13.353312 28661 rpcz_store.cc:276] 0503 14:12:12.296247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.449597 (+153350us) service_pool.cc:225] Handling call
0503 14:12:13.326641 (+877044us) ts_tablet_manager.cc:1120] Deleting tablet d7c79ffee3954b5c870f4a8aa6379517
0503 14:12:13.326649 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.326768 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":876695}
I20240503 14:12:13.353753 28604 tablet_bootstrap.cc:492] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:13.354435 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d7c79ffee3954b5c870f4a8aa6379517 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c79ffee3954b5c870f4a8aa6379517 already in progress: creating tablet
I20240503 14:12:13.354864 28661 tablet_replica.cc:330] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:13.355566 29423 raft_consensus.cc:482] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.356356 28661 raft_consensus.cc:2229] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:13.356762 28661 pending_rounds.cc:62] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:13.356949 28661 pending_rounds.cc:68] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597700758999040 op_type: NO_OP noop_request { }
I20240503 14:12:13.357214 28661 raft_consensus.cc:2866] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:13.357471 28661 raft_consensus.cc:2258] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.359189 28661 ts_tablet_manager.cc:1875] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.359486 28604 tablet_bootstrap.cc:654] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.355780 29423 raft_consensus.cc:504] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.373715 29423 leader_election.cc:288] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.375738 29427 tablet.cc:1722] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:13.377979 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50edc38cdbe74397aa4f4928d56d9fbc" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:13.381587 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50edc38cdbe74397aa4f4928d56d9fbc" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:13.387440 29406 raft_consensus.cc:504] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.388753 29406 leader_election.cc:288] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.389878 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265dc5f2630e453ea71abddb1a9753f3" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:13.390166 28491 raft_consensus.cc:3037] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:13.390194 28150 leader_election.cc:341] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:13.390503 28150 leader_election.cc:341] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:13.391402 28290 leader_election.cc:341] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 265dc5f2630e453ea71abddb1a9753f3
I20240503 14:12:13.394737 29424 consensus_queue.cc:1035] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:12:13.398383 28491 raft_consensus.cc:2454] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:13.399248 28150 leader_election.cc:302] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:13.401610 29426 tablet.cc:1722] T f03bf91cf5414208b6767482223bc240 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:13.404198 29281 raft_consensus.cc:2726] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:13.404493 28600 tablet_bootstrap.cc:654] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.408099 28214 raft_consensus.cc:2454] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:13.409396 28427 leader_election.cc:302] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:13.409761 29431 raft_consensus.cc:2781] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.409900 29431 raft_consensus.cc:686] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:13.410198 29431 consensus_queue.cc:237] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.416914 28604 tablet_bootstrap.cc:492] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
W20240503 14:12:13.417138 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9c0540960ac45aabeaf13f888caf238 consensus state has no leader
I20240503 14:12:13.417336 28026 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454] alter complete (version 2)
I20240503 14:12:13.417485 28604 ts_tablet_manager.cc:1367] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20240503 14:12:13.419313 28604 raft_consensus.cc:348] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.419608 28604 raft_consensus.cc:374] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.419692 28604 raft_consensus.cc:729] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.419648 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9622df66720f484b8f46a671ad60bf5d" } modify_external_catalogs: true
I20240503 14:12:13.421340 28604 consensus_queue.cc:260] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.424221 28604 ts_tablet_manager.cc:1398] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:13.431778 28352 raft_consensus.cc:2454] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:13.432502 28604 tablet_bootstrap.cc:492] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:13.434334 28661 ts_tablet_manager.cc:1888] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:13.435420 28661 log.cc:1196] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/605531ed29214b1ca27011c5f5e96b11
I20240503 14:12:13.436580 28661 ts_tablet_manager.cc:1909] T 605531ed29214b1ca27011c5f5e96b11 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:13.439975 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9622df66720f484b8f46a671ad60bf5d" } modify_external_catalogs: true
I20240503 14:12:13.441743 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a6c17cfce7e644668817bf3483a098d4
I20240503 14:12:13.441751 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 451) took 1142 ms. Trace:
I20240503 14:12:13.441936 28661 rpcz_store.cc:276] 0503 14:12:12.299671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.453431 (+153760us) service_pool.cc:225] Handling call
0503 14:12:13.354761 (+901330us) ts_tablet_manager.cc:1120] Deleting tablet 605531ed29214b1ca27011c5f5e96b11
0503 14:12:13.354774 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.362912 (+ 8138us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.433922 (+ 71010us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.441698 (+ 7776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":899859}
I20240503 14:12:13.438617 28604 tablet_bootstrap.cc:654] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.447997 28290 leader_election.cc:302] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:13.449925 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet 265dc5f2630e453ea71abddb1a9753f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:03 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.454463 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 448) took 1159 ms. Trace:
I20240503 14:12:13.454752 28661 rpcz_store.cc:276] 0503 14:12:12.295150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.447568 (+152418us) service_pool.cc:225] Handling call
0503 14:12:13.454143 (+1006575us) ts_tablet_manager.cc:1120] Deleting tablet 54f08aac9a7147ebaffda4262a57a86a
0503 14:12:13.454157 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.454377 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998897}
I20240503 14:12:13.455186 28661 tablet_replica.cc:330] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:13.455474 28661 raft_consensus.cc:2229] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.455696 28661 raft_consensus.cc:2258] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.456451 28661 ts_tablet_manager.cc:1875] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.459133 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 3514b79467dc42bc89fdacf8e39a000d from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:13.459945 29424 raft_consensus.cc:482] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.460203 29424 raft_consensus.cc:504] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.460448 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 605531ed29214b1ca27011c5f5e96b11 (table default.Table_2907862ac0af4c3692460a0071e8bce3 [id=651246fc28e144af9b8a0bbe9926aa22]) successfully deleted
I20240503 14:12:13.461236 29424 leader_election.cc:288] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.461407 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_021c165af8934008b2bb8bb0487c0af4" } modify_external_catalogs: true
I20240503 14:12:13.461699 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_021c165af8934008b2bb8bb0487c0af4" } modify_external_catalogs: true
W20240503 14:12:13.462554 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 consensus state has no leader
I20240503 14:12:13.463440 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_285ab08f9ab94ca6a053a091945582ae" } modify_external_catalogs: true
I20240503 14:12:13.463655 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_285ab08f9ab94ca6a053a091945582ae" } modify_external_catalogs: true
W20240503 14:12:13.463824 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 54f08aac9a7147ebaffda4262a57a86a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f08aac9a7147ebaffda4262a57a86a already in progress: creating tablet
I20240503 14:12:13.468806 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f41bb9c5d40b0913264864ecc3528" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:13.469660 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_808ffc4c24d0480b87dab400bd4aec7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:13.469964 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c17cfce7e644668817bf3483a098d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 10876f4512794e6aa72bda3c1da8b2d0) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:13.470391 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_808ffc4c24d0480b87dab400bd4aec7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:13.470789 28427 leader_election.cc:341] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.471747 29432 tablet.cc:1722] T f03bf91cf5414208b6767482223bc240 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:13.472469 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d467def5cd244a9a9069718e6070c314"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.473145 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d467def5cd244a9a9069718e6070c314 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.473433 29406 raft_consensus.cc:2781] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.473762 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet bd76060daab84c2f9f6727b8a9bb3a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fb79fcf30d485798c64c815e372dc1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.475199 28653 ts_tablet_manager.cc:1888] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.475404 28653 log.cc:1196] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a7295c2d3bd34e20a69986aadc9c7017
I20240503 14:12:13.476001 28653 ts_tablet_manager.cc:1909] T a7295c2d3bd34e20a69986aadc9c7017 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:13.467726 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9f5a767355f24f1b928c442018fe05c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:13.476531 28025 catalog_manager.cc:6554] ReplaceTablet: tablet a6c17cfce7e644668817bf3483a098d4 deleted and replaced by tablet 10876f4512794e6aa72bda3c1da8b2d0
I20240503 14:12:13.477121 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ff836f0ad0b34e63bf9ee17c3f678da8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:13.477748 28603 ts_tablet_manager.cc:1888] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.477952 28603 log.cc:1196] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/93462d2b1a474bd688d6266cd3b0695c
W20240503 14:12:13.478314 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff836f0ad0b34e63bf9ee17c3f678da8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.479621 28603 ts_tablet_manager.cc:1909] T 93462d2b1a474bd688d6266cd3b0695c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:13.480304 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f5a767355f24f1b928c442018fe05c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.481673 29456 raft_consensus.cc:482] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.481979 29456 raft_consensus.cc:504] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.483618 29456 leader_election.cc:288] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.484211 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 456) took 1027 ms. Trace:
I20240503 14:12:13.484357 28603 rpcz_store.cc:276] 0503 14:12:12.456549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.456608 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:13.167893 (+711285us) ts_tablet_manager.cc:1120] Deleting tablet 93462d2b1a474bd688d6266cd3b0695c
0503 14:12:13.167900 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.361330 (+193430us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.472798 (+111468us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.484146 (+ 11348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":710994}
I20240503 14:12:13.484268 29456 raft_consensus.cc:482] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.485428 29456 raft_consensus.cc:504] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.487177 28604 tablet_bootstrap.cc:492] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:13.487916 29456 leader_election.cc:288] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.488371 28604 ts_tablet_manager.cc:1367] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.004s
I20240503 14:12:13.484871 29406 raft_consensus.cc:686] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:13.488894 29406 consensus_queue.cc:237] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.484776 28603 tablet_replica.cc:330] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:13.489542 28603 raft_consensus.cc:2229] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.489789 28603 raft_consensus.cc:2258] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.489908 28604 raft_consensus.cc:348] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.490231 28604 raft_consensus.cc:374] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.490402 28604 raft_consensus.cc:729] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.490725 28604 consensus_queue.cc:260] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.491464 28604 ts_tablet_manager.cc:1398] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:13.491972 28604 tablet_bootstrap.cc:492] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:13.492777 28600 tablet_bootstrap.cc:492] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:13.493014 28600 ts_tablet_manager.cc:1367] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.381s user 0.002s sys 0.008s
I20240503 14:12:13.494132 28600 raft_consensus.cc:348] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.494455 28600 raft_consensus.cc:374] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.494573 28600 raft_consensus.cc:729] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.495486 28604 tablet_bootstrap.cc:654] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.496460 29295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:13.497027 29295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:13.497882 28614 tablet_bootstrap.cc:492] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:13.498087 28614 ts_tablet_manager.cc:1367] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.354s user 0.056s sys 0.070s
I20240503 14:12:13.499328 28614 raft_consensus.cc:348] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.499611 28614 raft_consensus.cc:374] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.499758 28614 raft_consensus.cc:729] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.500592 28026 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet ae909ab6263a401490952e3f68533741
I20240503 14:12:13.501083 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4fc8bf5e30814b5eb9e5ea28422b1802" } new_table_name: "default.Table_efd3e1b0423142738d51c2acc85076bc" modify_external_catalogs: true
I20240503 14:12:13.504033 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 7945118bb17b431dac8670007d2d0a18 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:13.504365 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd76060daab84c2f9f6727b8a9bb3a49" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:13.504763 28489 raft_consensus.cc:2454] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:13.505470 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:13.506057 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c17cfce7e644668817bf3483a098d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 10876f4512794e6aa72bda3c1da8b2d0) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:13.506369 28290 leader_election.cc:341] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: bd76060daab84c2f9f6727b8a9bb3a49
I20240503 14:12:13.506528 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f41bb9c5d40b0913264864ecc3528" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:13.506685 28290 leader_election.cc:302] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:13.507588 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda39f0323ec47799d3d44c96f3ba605" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:13.508037 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 93462d2b1a474bd688d6266cd3b0695c (table default.Table_7712b0c57d054073ac3f31fa183e72f1 [id=48c3f3e039044d10a6cfa1b6bcd5ae79]) successfully deleted
I20240503 14:12:13.508932 28290 leader_election.cc:302] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:13.509330 29456 raft_consensus.cc:2781] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.509430 29454 raft_consensus.cc:2726] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:13.509492 29456 raft_consensus.cc:482] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.509656 29456 raft_consensus.cc:3037] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.500010 28614 consensus_queue.cc:260] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.510258 28614 ts_tablet_manager.cc:1398] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:12:13.510641 28614 tablet_bootstrap.cc:492] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:13.513049 29456 raft_consensus.cc:504] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.513921 29456 leader_election.cc:288] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:13.514214 28427 leader_election.cc:341] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.514423 28427 leader_election.cc:302] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:13.514883 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 419) took 2219 ms. Trace:
I20240503 14:12:13.514989 29431 raft_consensus.cc:2726] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:13.515024 28653 rpcz_store.cc:276] 0503 14:12:11.294983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.357320 (+ 62337us) service_pool.cc:225] Handling call
0503 14:12:13.161481 (+1804161us) ts_tablet_manager.cc:1120] Deleting tablet a7295c2d3bd34e20a69986aadc9c7017
0503 14:12:13.161497 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.447084 (+285587us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.475043 (+ 27959us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.514810 (+ 39767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803808}
I20240503 14:12:13.515419 28604 tablet_bootstrap.cc:492] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:13.515691 28604 ts_tablet_manager.cc:1367] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.003s
I20240503 14:12:13.516239 28603 ts_tablet_manager.cc:1875] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.516880 28604 raft_consensus.cc:348] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.517247 28604 raft_consensus.cc:374] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.517251 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e19b96f79a6c41c0bc266f1391941564" } modify_external_catalogs: true
I20240503 14:12:13.517498 28604 raft_consensus.cc:729] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.517477 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e19b96f79a6c41c0bc266f1391941564" } modify_external_catalogs: true
I20240503 14:12:13.517894 28604 consensus_queue.cc:260] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.518611 28604 ts_tablet_manager.cc:1398] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:12:13.519003 28604 tablet_bootstrap.cc:492] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:13.519341 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_027b06bfe8c444dcba6cd5d6f42220ec" } new_table_name: "default.Table_713837a8f6e7403fa453e5203654186b" modify_external_catalogs: true
I20240503 14:12:13.521466 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c17cfce7e644668817bf3483a098d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 10876f4512794e6aa72bda3c1da8b2d0) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.521440 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8f9db6955d4e47c18e50ceefd3d230f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:13.494943 28600 consensus_queue.cc:260] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:13.522131 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f9db6955d4e47c18e50ceefd3d230f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.526129 29427 tablet.cc:1722] T f03bf91cf5414208b6767482223bc240 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 14:12:13.526369 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b94092725fa4956a4c00e3685f9aeaf consensus state has no leader
I20240503 14:12:13.526860 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_130e64b09fbf49ceac42c4f3bac8c0ba" } new_table_name: "default.Table_3ce7004b7c164629a92e6bcbb421348d" modify_external_catalogs: true
I20240503 14:12:13.532987 28600 ts_tablet_manager.cc:1398] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.040s user 0.003s sys 0.000s
I20240503 14:12:13.536160 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]). Expected version 2 got 0
I20240503 14:12:13.541484 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a7295c2d3bd34e20a69986aadc9c7017 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]) successfully deleted
I20240503 14:12:13.542094 28196 tablet_service.cc:1467] Processing CreateTablet for tablet dd3fca3ff7d94a8bb2a2ee15618fb76c (DEFAULT_TABLE table=default.Table_251eaa8fc8ff4652b02caa68687f4ec0 [id=fb7ab29934964ca4b667c32b279e9299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.542730 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3fca3ff7d94a8bb2a2ee15618fb76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.544530 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 5757e6e8703640519f7d38754cc0c8ce (DEFAULT_TABLE table=default.Table_3d53a12202f9477788f519d5ef0c9531 [id=50aaea9c426b48e4bd44a475a481088f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.544785 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.545128 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5757e6e8703640519f7d38754cc0c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.546756 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 13f8553cfbab4af2a38a73e81fed2a5f (DEFAULT_TABLE table=default.Table_33b986666be7452c8edcd5b0a0bf4f6f [id=35b27a312c774bcfb92ee037e6628f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.547315 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f8553cfbab4af2a38a73e81fed2a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.549193 28600 tablet_bootstrap.cc:492] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:13.545799 28191 tablet_service.cc:1467] Processing CreateTablet for tablet f54d1926578a4974bd561446fd47d89c (DEFAULT_TABLE table=default.Table_a5bf2e0d9bcc43b0bec8b4daf11b707b [id=e479f2d72dd14b058dc698c3de66c4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.554504 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54d1926578a4974bd561446fd47d89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.555183 28326 tablet_service.cc:1467] Processing CreateTablet for tablet cace51f8e5ff49f596c94f3698fdf9ef (DEFAULT_TABLE table=default.Table_90131726ecc0494f8e4eff1c20c6095f [id=f2e65b26dd874f1ea1c796b3733e1db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.555680 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cace51f8e5ff49f596c94f3698fdf9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.557408 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 13f8553cfbab4af2a38a73e81fed2a5f (DEFAULT_TABLE table=default.Table_33b986666be7452c8edcd5b0a0bf4f6f [id=35b27a312c774bcfb92ee037e6628f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.557963 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f8553cfbab4af2a38a73e81fed2a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.561041 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 3b435cb73d3e4c9fbeadaf0d54b2bd4f (DEFAULT_TABLE table=default.Table_46ee005206eb49879a8df1d11d8d331c [id=30cd5395ae4046bf809efb4f164832bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.561698 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b435cb73d3e4c9fbeadaf0d54b2bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.566955 28318 tablet_service.cc:1467] Processing CreateTablet for tablet f54d1926578a4974bd561446fd47d89c (DEFAULT_TABLE table=default.Table_a5bf2e0d9bcc43b0bec8b4daf11b707b [id=e479f2d72dd14b058dc698c3de66c4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.567550 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54d1926578a4974bd561446fd47d89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.568356 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd76060daab84c2f9f6727b8a9bb3a49" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:13.568655 28491 raft_consensus.cc:3037] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.569411 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 5757e6e8703640519f7d38754cc0c8ce (DEFAULT_TABLE table=default.Table_3d53a12202f9477788f519d5ef0c9531 [id=50aaea9c426b48e4bd44a475a481088f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.570060 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5757e6e8703640519f7d38754cc0c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.573937 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 34fa067359a640eeba8c80d08371904e
I20240503 14:12:13.574744 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 57ab077988364bd687e54f90c3c265d5 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:13.574836 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c7e5a76bf36545a5b53fc3d60512a521" } modify_external_catalogs: true
I20240503 14:12:13.575114 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c7e5a76bf36545a5b53fc3d60512a521" } modify_external_catalogs: true
I20240503 14:12:13.575930 28603 ts_tablet_manager.cc:1888] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.576143 28603 log.cc:1196] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/129717ef137c41c5a08d401c64164187
I20240503 14:12:13.576506 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3514b79467dc42bc89fdacf8e39a000d
I20240503 14:12:13.576645 28603 ts_tablet_manager.cc:1909] T 129717ef137c41c5a08d401c64164187 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:13.577039 28033 catalog_manager.cc:6554] ReplaceTablet: tablet 3514b79467dc42bc89fdacf8e39a000d deleted and replaced by tablet dee5a230ef584077ba9f4a27c878ef4d
I20240503 14:12:13.577369 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet ae1cd28bc07e4ccd9097f6dee471c5c6 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:13.577550 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 457) took 1120 ms. Trace:
I20240503 14:12:13.577668 28603 rpcz_store.cc:276] 0503 14:12:12.457189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.457239 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:13.484697 (+1027458us) ts_tablet_manager.cc:1120] Deleting tablet 129717ef137c41c5a08d401c64164187
0503 14:12:13.484709 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.562026 (+ 77317us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.575790 (+ 13764us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.577478 (+ 1688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027202}
I20240503 14:12:13.578223 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 466) took 1025 ms. Trace:
I20240503 14:12:13.578337 28603 rpcz_store.cc:276] 0503 14:12:12.552499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.554405 (+ 1906us) service_pool.cc:225] Handling call
0503 14:12:13.577979 (+1023574us) ts_tablet_manager.cc:1120] Deleting tablet 007a892ef8504b848a400c05740eaa0c
0503 14:12:13.577990 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.578126 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019977}
W20240503 14:12:13.579620 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69425d29d55b4fe4bb7f08ae390aaa86 consensus state has no leader
I20240503 14:12:13.580034 28603 tablet_replica.cc:330] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:13.581138 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 16fed718124a4640967ab13a1ac2b379 (DEFAULT_TABLE table=default.Table_c89fe31dfbac4d48a061a21a358b84f6 [id=4d7f7a3f533b469fa179c7821ca561b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.581696 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fed718124a4640967ab13a1ac2b379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.583344 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 5757e6e8703640519f7d38754cc0c8ce (DEFAULT_TABLE table=default.Table_3d53a12202f9477788f519d5ef0c9531 [id=50aaea9c426b48e4bd44a475a481088f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:13.584149 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed4dc732bcb44222b4b15e5d77defe33 consensus state has no leader
I20240503 14:12:13.588433 28457 tablet_service.cc:1467] Processing CreateTablet for tablet cace51f8e5ff49f596c94f3698fdf9ef (DEFAULT_TABLE table=default.Table_90131726ecc0494f8e4eff1c20c6095f [id=f2e65b26dd874f1ea1c796b3733e1db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.589895 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fa067359a640eeba8c80d08371904e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.590229 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 3514b79467dc42bc89fdacf8e39a000d with delete_type TABLET_DATA_DELETED (replaced by tablet dee5a230ef584077ba9f4a27c878ef4d) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.590179 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 13f8553cfbab4af2a38a73e81fed2a5f (DEFAULT_TABLE table=default.Table_33b986666be7452c8edcd5b0a0bf4f6f [id=35b27a312c774bcfb92ee037e6628f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.590687 29431 raft_consensus.cc:482] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.590893 29431 raft_consensus.cc:504] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.583563 28193 tablet_service.cc:1467] Processing CreateTablet for tablet cace51f8e5ff49f596c94f3698fdf9ef (DEFAULT_TABLE table=default.Table_90131726ecc0494f8e4eff1c20c6095f [id=f2e65b26dd874f1ea1c796b3733e1db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.591797 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cace51f8e5ff49f596c94f3698fdf9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.533607 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 423) took 2015 ms. Trace:
I20240503 14:12:13.596148 28458 tablet_service.cc:1467] Processing CreateTablet for tablet f54d1926578a4974bd561446fd47d89c (DEFAULT_TABLE table=default.Table_a5bf2e0d9bcc43b0bec8b4daf11b707b [id=e479f2d72dd14b058dc698c3de66c4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.596414 28491 raft_consensus.cc:2454] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:13.596482 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 60206e8e73e84e409ed6aba222d9ff4b
I20240503 14:12:13.598290 29431 leader_election.cc:288] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.598362 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 3b435cb73d3e4c9fbeadaf0d54b2bd4f (DEFAULT_TABLE table=default.Table_46ee005206eb49879a8df1d11d8d331c [id=30cd5395ae4046bf809efb4f164832bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.598675 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f8553cfbab4af2a38a73e81fed2a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.599123 28460 tablet_service.cc:1467] Processing CreateTablet for tablet dd3fca3ff7d94a8bb2a2ee15618fb76c (DEFAULT_TABLE table=default.Table_251eaa8fc8ff4652b02caa68687f4ec0 [id=fb7ab29934964ca4b667c32b279e9299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.600240 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3fca3ff7d94a8bb2a2ee15618fb76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.602919 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fa067359a640eeba8c80d08371904e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.603170 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5757e6e8703640519f7d38754cc0c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.604293 28604 tablet_bootstrap.cc:654] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.604861 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 16fed718124a4640967ab13a1ac2b379 (DEFAULT_TABLE table=default.Table_c89fe31dfbac4d48a061a21a358b84f6 [id=4d7f7a3f533b469fa179c7821ca561b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.605410 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.611047 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cace51f8e5ff49f596c94f3698fdf9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.611438 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 3514b79467dc42bc89fdacf8e39a000d with delete_type TABLET_DATA_DELETED (replaced by tablet dee5a230ef584077ba9f4a27c878ef4d) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.585877 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 3b435cb73d3e4c9fbeadaf0d54b2bd4f (DEFAULT_TABLE table=default.Table_46ee005206eb49879a8df1d11d8d331c [id=30cd5395ae4046bf809efb4f164832bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.586956 28321 tablet_service.cc:1467] Processing CreateTablet for tablet dd3fca3ff7d94a8bb2a2ee15618fb76c (DEFAULT_TABLE table=default.Table_251eaa8fc8ff4652b02caa68687f4ec0 [id=fb7ab29934964ca4b667c32b279e9299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.612202 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54d1926578a4974bd561446fd47d89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.587424 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 16fed718124a4640967ab13a1ac2b379 (DEFAULT_TABLE table=default.Table_c89fe31dfbac4d48a061a21a358b84f6 [id=4d7f7a3f533b469fa179c7821ca561b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.587842 28330 tablet_service.cc:1467] Processing CreateTablet for tablet c04707221b5241db8cd380ad5db24af5 (DEFAULT_TABLE table=default.Table_2795914729c948ed94f38a47c734d74d [id=ffddf25be2624109aa50aa247edb4b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.588184 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fa067359a640eeba8c80d08371904e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.588416 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 3514b79467dc42bc89fdacf8e39a000d with delete_type TABLET_DATA_DELETED (replaced by tablet dee5a230ef584077ba9f4a27c878ef4d) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.612372 28603 raft_consensus.cc:2229] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.597286 28352 raft_consensus.cc:3037] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.612711 28603 raft_consensus.cc:2258] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.613107 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fed718124a4640967ab13a1ac2b379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.613646 28603 ts_tablet_manager.cc:1875] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.614316 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b435cb73d3e4c9fbeadaf0d54b2bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.619676 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3f6ca1c2e7314e7ebb2235561072ed1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.620402 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f6ca1c2e7314e7ebb2235561072ed1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.621660 28603 ts_tablet_manager.cc:1888] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:13.621842 28603 log.cc:1196] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/651478ef632144079f93f3b50bff4611
I20240503 14:12:13.622381 28603 ts_tablet_manager.cc:1909] T 651478ef632144079f93f3b50bff4611 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:13.624042 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.624274 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cc8ca3c970a949fab24298af4538c323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.625038 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc8ca3c970a949fab24298af4538c323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.625114 29431 raft_consensus.cc:482] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.625420 29431 raft_consensus.cc:504] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.625468 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9f6a864cf34b498f87ccf5afca7d29e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.626073 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f6a864cf34b498f87ccf5afca7d29e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.626307 29431 leader_election.cc:288] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.631053 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3703ab7db61b4c9ead3058bfdc123189" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:13.598150 28614 tablet_bootstrap.cc:654] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.632316 28212 raft_consensus.cc:2454] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:13.601963 28290 leader_election.cc:302] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:13.638378 29455 raft_consensus.cc:2781] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.638641 29455 raft_consensus.cc:686] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:13.639110 29455 consensus_queue.cc:237] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:13.638399 28290 leader_election.cc:341] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: bd76060daab84c2f9f6727b8a9bb3a49
I20240503 14:12:13.650187 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.650660 28600 tablet_bootstrap.cc:654] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.652411 29281 raft_consensus.cc:482] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.652623 29281 raft_consensus.cc:504] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.653466 29281 leader_election.cc:288] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.653903 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fdf88fefa4ff7939f3ca05a82d7ba" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:13.655222 28212 raft_consensus.cc:2454] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:13.655198 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6627f0d9a544bb8176bea6c1af5ab4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:13.655937 28150 leader_election.cc:341] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.656365 29431 consensus_queue.cc:1035] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:13.657989 28661 ts_tablet_manager.cc:1888] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:13.658851 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 54577.6us
I20240503 14:12:13.659937 28427 leader_election.cc:302] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:13.664350 29431 raft_consensus.cc:2781] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.667955 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6627f0d9a544bb8176bea6c1af5ab4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:13.668145 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fdf88fefa4ff7939f3ca05a82d7ba" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:13.668493 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3703ab7db61b4c9ead3058bfdc123189" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:13.669111 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b435cb73d3e4c9fbeadaf0d54b2bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:13.669139 28150 leader_election.cc:341] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.669308 28150 leader_election.cc:302] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:13.669719 28603 tablet_replica.cc:330] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:13.669798 29281 raft_consensus.cc:2726] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:13.669989 28603 raft_consensus.cc:2229] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:13.670310 28603 pending_rounds.cc:62] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:13.670444 28603 pending_rounds.cc:68] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597705181577216 op_type: NO_OP noop_request { }
I20240503 14:12:13.670591 28603 raft_consensus.cc:2866] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:13.670742 28603 raft_consensus.cc:2258] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.671196 28661 log.cc:1196] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/129717ef137c41c5a08d401c64164187
I20240503 14:12:13.672283 28661 ts_tablet_manager.cc:1909] T 129717ef137c41c5a08d401c64164187 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:13.673171 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16fed718124a4640967ab13a1ac2b379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.673246 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 455) took 1219 ms. Trace:
I20240503 14:12:13.673375 28661 rpcz_store.cc:276] 0503 14:12:12.454085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.456645 (+ 2560us) service_pool.cc:225] Handling call
0503 14:12:13.455082 (+998437us) ts_tablet_manager.cc:1120] Deleting tablet 129717ef137c41c5a08d401c64164187
0503 14:12:13.455094 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.522879 (+ 67785us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.602124 (+ 79245us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.673142 (+ 71018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998165}
I20240503 14:12:13.673817 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04707221b5241db8cd380ad5db24af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.674453 28603 ts_tablet_manager.cc:1875] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.603994 28653 rpcz_store.cc:276] 0503 14:12:11.518162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.518244 (+ 82us) service_pool.cc:225] Handling call
0503 14:12:13.533325 (+2015081us) ts_tablet_manager.cc:1120] Deleting tablet dbcc6cef7b7d464a9fbbae61caf0437d
0503 14:12:13.533339 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.533509 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2014646}
I20240503 14:12:13.675312 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3fca3ff7d94a8bb2a2ee15618fb76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.675624 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 129717ef137c41c5a08d401c64164187 (table default.Table_36c01c2a0d944899bf662092e1cc816d [id=f9add48fee4b4ca89754d81057f832e4]) successfully deleted
W20240503 14:12:13.676137 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b476e87faea1415d95ae014a93fa22af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b476e87faea1415d95ae014a93fa22af already in progress: creating tablet
W20240503 14:12:13.676383 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d4c8a97076a44f9eaa3a15026f90a304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c8a97076a44f9eaa3a15026f90a304 already in progress: creating tablet
W20240503 14:12:13.676630 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7b6c998e518541248a87e4c1551ec080 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b6c998e518541248a87e4c1551ec080 already in progress: creating tablet
W20240503 14:12:13.676829 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ce4846d662a64d51b16fd5447c371e9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce4846d662a64d51b16fd5447c371e9a already in progress: creating tablet
W20240503 14:12:13.677047 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 320c526e845e4dc78da345bcb095918f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 320c526e845e4dc78da345bcb095918f already in progress: creating tablet
W20240503 14:12:13.677227 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 4084bd4fbf604042bb33ff03c3f8cc19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4084bd4fbf604042bb33ff03c3f8cc19 already in progress: creating tablet
W20240503 14:12:13.677441 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ca6627f0d9a544bb8176bea6c1af5ab4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca6627f0d9a544bb8176bea6c1af5ab4 already in progress: creating tablet
I20240503 14:12:13.678644 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 424) took 2152 ms. Trace:
I20240503 14:12:13.678776 28653 rpcz_store.cc:276] 0503 14:12:11.526048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.526114 (+ 66us) service_pool.cc:225] Handling call
0503 14:12:13.678378 (+2152264us) ts_tablet_manager.cc:1120] Deleting tablet 8410e8b6b511403ea609b1353855937b
0503 14:12:13.678393 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.678553 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2152002}
I20240503 14:12:13.680200 28653 tablet_replica.cc:330] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:13.680450 28653 raft_consensus.cc:2229] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.680752 28653 raft_consensus.cc:2258] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.680665 29431 raft_consensus.cc:482] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.681169 29431 raft_consensus.cc:3037] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.681404 28653 ts_tablet_manager.cc:1875] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.682977 28195 tablet_service.cc:1467] Processing CreateTablet for tablet c04707221b5241db8cd380ad5db24af5 (DEFAULT_TABLE table=default.Table_2795914729c948ed94f38a47c734d74d [id=ffddf25be2624109aa50aa247edb4b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.683750 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04707221b5241db8cd380ad5db24af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.684701 28427 leader_election.cc:302] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:13.687765 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 39bf3cc9eda24c99815ddcec79b1ff78 (DEFAULT_TABLE table=default.Table_2b685af443594f2f93b99eb4a1df3e13 [id=306349082c434ed6861cc8999b267e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.688300 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bf3cc9eda24c99815ddcec79b1ff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.688793 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 9aaf7d4927674eedbfd486df6a347f7c (DEFAULT_TABLE table=default.Table_ec3edf11cc5746799ddb3623736b23fe [id=957bf351cc3c4161b4ddc0dcab2f963d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.689321 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aaf7d4927674eedbfd486df6a347f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.689306 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 94ec7c7c9a424240a2a105e8bd9fc05e (DEFAULT_TABLE table=default.Table_a0200d3cb5e84132adf59bc5c3241d88 [id=df47a2d37d1b4ebea1ca4bbc54237bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.689996 29424 raft_consensus.cc:2781] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.690171 29424 raft_consensus.cc:482] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.690331 29424 raft_consensus.cc:3037] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:13.693112 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet dbcc6cef7b7d464a9fbbae61caf0437d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbcc6cef7b7d464a9fbbae61caf0437d already in progress: creating tablet
W20240503 14:12:13.693598 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 8410e8b6b511403ea609b1353855937b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8410e8b6b511403ea609b1353855937b already in progress: creating tablet
I20240503 14:12:13.694883 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ec7c7c9a424240a2a105e8bd9fc05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.697865 29424 raft_consensus.cc:504] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.698660 28614 tablet_bootstrap.cc:492] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:13.698932 28614 ts_tablet_manager.cc:1367] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.188s user 0.000s sys 0.006s
I20240503 14:12:13.699965 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7945118bb17b431dac8670007d2d0a18
I20240503 14:12:13.700361 29424 leader_election.cc:288] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.700445 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f8fe462c9ee54a27b9714f6457036d76
I20240503 14:12:13.703631 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:13.704450 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 57ab077988364bd687e54f90c3c265d5
I20240503 14:12:13.704732 29431 raft_consensus.cc:504] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.705049 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 57ab077988364bd687e54f90c3c265d5 deleted and replaced by tablet 8cd037e135904e2cab09cbf39fed0b13
I20240503 14:12:13.705660 29431 leader_election.cc:288] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.705572 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a0b97d2f3edd4c549bbd9c1748f4d82f" } new_table_name: "default.Table_4852bd62507141ed994fa106a0927878" modify_external_catalogs: true
I20240503 14:12:13.706954 28600 tablet_bootstrap.cc:492] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:13.707156 28600 ts_tablet_manager.cc:1367] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.166s user 0.003s sys 0.003s
I20240503 14:12:13.707407 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ae1cd28bc07e4ccd9097f6dee471c5c6
I20240503 14:12:13.707926 28033 catalog_manager.cc:6554] ReplaceTablet: tablet ae1cd28bc07e4ccd9097f6dee471c5c6 deleted and replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c
I20240503 14:12:13.708415 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d4cc46240b5f4f88bbf7c96f7a78d0c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.709118 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4cc46240b5f4f88bbf7c96f7a78d0c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.709097 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 94ec7c7c9a424240a2a105e8bd9fc05e (DEFAULT_TABLE table=default.Table_a0200d3cb5e84132adf59bc5c3241d88 [id=df47a2d37d1b4ebea1ca4bbc54237bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.709611 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ec7c7c9a424240a2a105e8bd9fc05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.718215 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 7945118bb17b431dac8670007d2d0a18 deleted and replaced by tablet b0993be031cc4792a9eadba9ce41d024
I20240503 14:12:13.718766 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7e0f5faa00554a3b9cdaff8e346d048a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.719513 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e0f5faa00554a3b9cdaff8e346d048a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.723081 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ac752d8ab4fd4a218b052f388f416494
I20240503 14:12:13.723924 28603 ts_tablet_manager.cc:1888] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:13.724045 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d7bd421e71724c8a89b7cd478f447a3b" } new_table_name: "default.Table_8894e326ff3a419d82f7e466b85bf687" modify_external_catalogs: true
I20240503 14:12:13.725158 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 39bf3cc9eda24c99815ddcec79b1ff78 (DEFAULT_TABLE table=default.Table_2b685af443594f2f93b99eb4a1df3e13 [id=306349082c434ed6861cc8999b267e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.725829 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 9aaf7d4927674eedbfd486df6a347f7c (DEFAULT_TABLE table=default.Table_ec3edf11cc5746799ddb3623736b23fe [id=957bf351cc3c4161b4ddc0dcab2f963d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.699952 28614 raft_consensus.cc:348] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.726709 28614 raft_consensus.cc:374] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.726835 28614 raft_consensus.cc:729] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.726984 28603 log.cc:1196] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/265dc5f2630e453ea71abddb1a9753f3
I20240503 14:12:13.727926 28603 ts_tablet_manager.cc:1909] T 265dc5f2630e453ea71abddb1a9753f3 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:13.729686 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 456) took 1260 ms. Trace:
I20240503 14:12:13.729938 28600 raft_consensus.cc:348] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.727159 28614 consensus_queue.cc:260] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:13.730628 28427 leader_election.cc:341] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.731410 28614 ts_tablet_manager.cc:1398] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20240503 14:12:13.729955 28661 rpcz_store.cc:276] 0503 14:12:12.456066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.461774 (+ 5708us) service_pool.cc:225] Handling call
0503 14:12:13.673750 (+1211976us) ts_tablet_manager.cc:1120] Deleting tablet c816f3b05cfb4496984f6cb28d310cc4
0503 14:12:13.673760 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.716927 (+ 43167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211723}
W20240503 14:12:13.731911 28427 leader_election.cc:341] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.732262 28460 tablet_service.cc:1467] Processing CreateTablet for tablet c04707221b5241db8cd380ad5db24af5 (DEFAULT_TABLE table=default.Table_2795914729c948ed94f38a47c734d74d [id=ffddf25be2624109aa50aa247edb4b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.735073 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 129717ef137c41c5a08d401c64164187 (table default.Table_36c01c2a0d944899bf662092e1cc816d [id=f9add48fee4b4ca89754d81057f832e4]) successfully deleted
I20240503 14:12:13.735528 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]) successfully deleted
I20240503 14:12:13.736668 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fdf88fefa4ff7939f3ca05a82d7ba" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:13.736783 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3703ab7db61b4c9ead3058bfdc123189" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:13.737587 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 94ec7c7c9a424240a2a105e8bd9fc05e (DEFAULT_TABLE table=default.Table_a0200d3cb5e84132adf59bc5c3241d88 [id=df47a2d37d1b4ebea1ca4bbc54237bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.737325 28334 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fe462c9ee54a27b9714f6457036d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.740149 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bf3cc9eda24c99815ddcec79b1ff78. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:13.743826 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c816f3b05cfb4496984f6cb28d310cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c816f3b05cfb4496984f6cb28d310cc4 already in progress: creating tablet
W20240503 14:12:13.748694 28427 leader_election.cc:341] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.734361 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fdf88fefa4ff7939f3ca05a82d7ba" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:13.735320 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3703ab7db61b4c9ead3058bfdc123189" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:13.747130 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet ac752d8ab4fd4a218b052f388f416494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.747284 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.750176 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aaf7d4927674eedbfd486df6a347f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.750595 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 39bf3cc9eda24c99815ddcec79b1ff78 (DEFAULT_TABLE table=default.Table_2b685af443594f2f93b99eb4a1df3e13 [id=306349082c434ed6861cc8999b267e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.751628 29431 raft_consensus.cc:482] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.752027 29431 raft_consensus.cc:504] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.753705 29431 leader_election.cc:288] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:13.754755 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 9aaf7d4927674eedbfd486df6a347f7c (DEFAULT_TABLE table=default.Table_ec3edf11cc5746799ddb3623736b23fe [id=957bf351cc3c4161b4ddc0dcab2f963d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.755719 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ab077988364bd687e54f90c3c265d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cd037e135904e2cab09cbf39fed0b13) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.755820 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aaf7d4927674eedbfd486df6a347f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.756978 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_28c16c4a952d45c0b34fd1d4864ac748" } modify_external_catalogs: true
I20240503 14:12:13.757294 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_28c16c4a952d45c0b34fd1d4864ac748" } modify_external_catalogs: true
I20240503 14:12:13.758482 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.759109 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bf3cc9eda24c99815ddcec79b1ff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.759426 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 7945118bb17b431dac8670007d2d0a18 with delete_type TABLET_DATA_DELETED (replaced by tablet b0993be031cc4792a9eadba9ce41d024) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.759651 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d4a44441aa4647449f447922587e6ffd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.760403 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4a44441aa4647449f447922587e6ffd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:13.760457 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d832cd30f2414303818d34a0cdf47e60 consensus state has no leader
I20240503 14:12:13.760454 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ec7c7c9a424240a2a105e8bd9fc05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.760990 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet ac752d8ab4fd4a218b052f388f416494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.761073 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_738df9de21164980b362ce19f71f3d9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:13.761183 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d0e52f75ad7f415bac0142d93ff4aa78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.761724 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_738df9de21164980b362ce19f71f3d9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.763664 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0ddd72e685204e87a5edb2761dda0b0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:13.764163 28603 tablet_replica.cc:330] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
W20240503 14:12:13.764425 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ddd72e685204e87a5edb2761dda0b0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.764575 28603 raft_consensus.cc:2229] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:13.764922 28603 pending_rounds.cc:62] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:13.765065 28603 pending_rounds.cc:68] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597705789702144 op_type: NO_OP noop_request { }
I20240503 14:12:13.765255 28603 raft_consensus.cc:2866] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:13.765425 28603 raft_consensus.cc:2258] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.766287 28614 tablet_bootstrap.cc:492] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:13.767287 28427 leader_election.cc:341] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:13.769147 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31ddd791d4e14630bfb3b4b1513f76e1 consensus state has no leader
I20240503 14:12:13.769742 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_272609e488f340d29926451ba0376de0" } new_table_name: "default.Table_4d92feade7db42d483f2f56f9788f20f" modify_external_catalogs: true
I20240503 14:12:13.770211 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04707221b5241db8cd380ad5db24af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.772645 28031 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_ac2a630ab8d046f88a48f486ed63dc80 [id=ddcf6334b4fc421f9abe196844fa69fc] alter complete (version 1)
I20240503 14:12:13.774106 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 465) took 1182 ms. Trace:
I20240503 14:12:13.779186 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f08aac9a7147ebaffda4262a57a86a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:13.743245 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ab077988364bd687e54f90c3c265d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cd037e135904e2cab09cbf39fed0b13) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.780545 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 7945118bb17b431dac8670007d2d0a18 with delete_type TABLET_DATA_DELETED (replaced by tablet b0993be031cc4792a9eadba9ce41d024) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.781368 28212 raft_consensus.cc:3037] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.784621 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_eeca27b59dd543faa1ea8c60944ddadf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.785460 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eeca27b59dd543faa1ea8c60944ddadf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.787120 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_40cc648a463747e68942c60a9d687deb" } modify_external_catalogs: true
I20240503 14:12:13.787401 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_40cc648a463747e68942c60a9d687deb" } modify_external_catalogs: true
I20240503 14:12:13.787886 28215 raft_consensus.cc:3037] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.790808 28603 ts_tablet_manager.cc:1875] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.791261 28027 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:13.791347 28212 raft_consensus.cc:2454] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:13.792701 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a2f9a3630d1d4bb79f9389f8e58257f5" } new_table_name: "default.Table_867e9a6baa014dada3c5c1ff572ab941" modify_external_catalogs: true
I20240503 14:12:13.793768 28614 tablet_bootstrap.cc:654] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:13.761741 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0e52f75ad7f415bac0142d93ff4aa78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.796672 28600 raft_consensus.cc:374] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.803459 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fe462c9ee54a27b9714f6457036d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.803642 28603 ts_tablet_manager.cc:1888] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:13.803841 28603 log.cc:1196] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/bd76060daab84c2f9f6727b8a9bb3a49
I20240503 14:12:13.804610 28603 ts_tablet_manager.cc:1909] T bd76060daab84c2f9f6727b8a9bb3a49 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:13.805564 28603 tablet_replica.cc:330] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:13.806062 28603 raft_consensus.cc:2229] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.806309 28603 raft_consensus.cc:2258] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.809110 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0f214d103abd4357829af22666b5f9a7" } new_table_name: "default.Table_9c932c51a0a844c38b521b84c50940a8" modify_external_catalogs: true
I20240503 14:12:13.812536 28603 ts_tablet_manager.cc:1875] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:13.815441 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 61624.2us
I20240503 14:12:13.816519 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.816596 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e977f7837e49fb93ef0fd18b619792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.817234 28600 raft_consensus.cc:729] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.817677 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 265dc5f2630e453ea71abddb1a9753f3 (table default.Table_cba5fd0a90c8433980631ba7e2048d8b [id=3368a0430dc94c4db0ae6260f5a0acf0]) successfully deleted
I20240503 14:12:13.818305 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet bd76060daab84c2f9f6727b8a9bb3a49 (table default.Table_3668987e06c74fa6bc566d6e46f92879 [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
I20240503 14:12:13.817494 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f08aac9a7147ebaffda4262a57a86a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:13.817814 29281 raft_consensus.cc:482] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:13.818590 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a6c17cfce7e644668817bf3483a098d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c17cfce7e644668817bf3483a098d4 already in progress: creating tablet
I20240503 14:12:13.818642 29281 raft_consensus.cc:504] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:13.818876 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 34fa067359a640eeba8c80d08371904e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34fa067359a640eeba8c80d08371904e already in progress: creating tablet
I20240503 14:12:13.819454 29281 leader_election.cc:288] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.820365 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514b79467dc42bc89fdacf8e39a000d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:13.821044 28150 leader_election.cc:341] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:13.821560 28212 raft_consensus.cc:2454] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:13.817982 28215 raft_consensus.cc:2454] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:13.818320 28600 consensus_queue.cc:260] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.825652 28614 tablet_bootstrap.cc:492] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:13.824841 28661 rpcz_store.cc:276] 0503 14:12:12.549379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.554238 (+ 4859us) service_pool.cc:225] Handling call
0503 14:12:13.731929 (+1177691us) ts_tablet_manager.cc:1120] Deleting tablet 007a892ef8504b848a400c05740eaa0c
0503 14:12:13.731945 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.732191 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175736}
I20240503 14:12:13.825878 28614 ts_tablet_manager.cc:1367] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.004s
I20240503 14:12:13.826953 28614 raft_consensus.cc:348] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.827287 28614 raft_consensus.cc:374] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.827494 28614 raft_consensus.cc:729] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.827786 28614 consensus_queue.cc:260] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.828511 28614 ts_tablet_manager.cc:1398] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:13.828920 28614 tablet_bootstrap.cc:492] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:13.831324 28427 leader_election.cc:302] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:13.832047 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet ac752d8ab4fd4a218b052f388f416494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.832430 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.835090 28614 tablet_bootstrap.cc:654] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.836295 28603 ts_tablet_manager.cc:1888] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.836959 28603 log.cc:1196] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/3514b79467dc42bc89fdacf8e39a000d
I20240503 14:12:13.837531 28603 ts_tablet_manager.cc:1909] T 3514b79467dc42bc89fdacf8e39a000d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:13.838317 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 3514b79467dc42bc89fdacf8e39a000d (table default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]) successfully deleted
I20240503 14:12:13.839632 28603 tablet_replica.cc:330] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:13.839916 28603 raft_consensus.cc:2229] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.840256 28603 raft_consensus.cc:2258] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.840986 28603 ts_tablet_manager.cc:1875] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.844518 29431 raft_consensus.cc:2781] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.848572 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 920b41e3b21347b393497de4d4e87469
I20240503 14:12:13.849452 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet aa3ce05b422a4166bfac468acb84e4cc from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:13.853320 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.865028 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:13.869050 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_efd3e1b0423142738d51c2acc85076bc" } modify_external_catalogs: true
I20240503 14:12:13.869345 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_efd3e1b0423142738d51c2acc85076bc" } modify_external_catalogs: true
W20240503 14:12:13.870486 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 788e4bfa1e414630939bc19da1ede3fb consensus state has no leader
W20240503 14:12:13.871101 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 888636e9beca43edab96a8736751409e consensus state has no leader
W20240503 14:12:13.871955 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02dd4c914a4846bab3a00408247ec64a consensus state has no leader
I20240503 14:12:13.873162 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9b21cd1262d94d1c994b29c44f86cb5e
I20240503 14:12:13.876358 28427 leader_election.cc:302] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:13.876690 28600 ts_tablet_manager.cc:1398] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.148s user 0.001s sys 0.001s
W20240503 14:12:13.876710 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 44843.8us
I20240503 14:12:13.880425 28600 tablet_bootstrap.cc:492] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:13.883028 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fe462c9ee54a27b9714f6457036d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.883482 28661 tablet_replica.cc:330] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:13.883445 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 44717.5us
I20240503 14:12:13.883749 28661 raft_consensus.cc:2229] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.884017 28661 raft_consensus.cc:2258] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.883319 29424 raft_consensus.cc:2781] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.884492 29424 raft_consensus.cc:482] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.884635 29424 raft_consensus.cc:3037] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.884738 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9262a7e828489ab51a745068c737df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.884927 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b41e3b21347b393497de4d4e87469 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.885372 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b41e3b21347b393497de4d4e87469 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.885438 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.885696 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.848987 28603 ts_tablet_manager.cc:1888] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.892289 28603 log.cc:1196] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7945118bb17b431dac8670007d2d0a18
I20240503 14:12:13.883098 28427 leader_election.cc:302] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:13.892884 28603 ts_tablet_manager.cc:1909] T 7945118bb17b431dac8670007d2d0a18 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:13.893126 28427 leader_election.cc:341] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.894345 28603 tablet_replica.cc:330] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
W20240503 14:12:13.894371 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 60206e8e73e84e409ed6aba222d9ff4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60206e8e73e84e409ed6aba222d9ff4b already in progress: creating tablet
I20240503 14:12:13.894623 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b41e3b21347b393497de4d4e87469 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.894891 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.895228 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.895480 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514b79467dc42bc89fdacf8e39a000d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:13.895756 28491 raft_consensus.cc:2454] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:13.896066 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 7945118bb17b431dac8670007d2d0a18 with delete_type TABLET_DATA_DELETED (replaced by tablet b0993be031cc4792a9eadba9ce41d024) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.884330 28604 tablet_bootstrap.cc:492] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:13.896317 29317 raft_consensus.cc:2781] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.896399 28604 ts_tablet_manager.cc:1367] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.377s user 0.010s sys 0.007s
I20240503 14:12:13.896507 29317 raft_consensus.cc:686] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:13.896896 29317 consensus_queue.cc:237] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:13.894747 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f8fe462c9ee54a27b9714f6457036d76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8fe462c9ee54a27b9714f6457036d76 already in progress: creating tablet
I20240503 14:12:13.897459 28604 raft_consensus.cc:348] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.897703 28604 raft_consensus.cc:374] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.897830 28604 raft_consensus.cc:729] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:13.898023 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 57ab077988364bd687e54f90c3c265d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57ab077988364bd687e54f90c3c265d5 already in progress: creating tablet
I20240503 14:12:13.898145 28604 consensus_queue.cc:260] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:13.898398 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae1cd28bc07e4ccd9097f6dee471c5c6 already in progress: creating tablet
I20240503 14:12:13.898558 28150 leader_election.cc:302] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:13.898655 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7945118bb17b431dac8670007d2d0a18 (table default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]) successfully deleted
W20240503 14:12:13.898932 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ac752d8ab4fd4a218b052f388f416494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac752d8ab4fd4a218b052f388f416494 already in progress: creating tablet
I20240503 14:12:13.899202 29281 raft_consensus.cc:2781] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:13.899365 29281 raft_consensus.cc:482] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.899477 29281 raft_consensus.cc:3037] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.900064 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd5234791524eddb7edd493e39fb993 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b21cd1262d94d1c994b29c44f86cb5e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.902416 28604 ts_tablet_manager.cc:1398] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:13.902776 28604 tablet_bootstrap.cc:492] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:13.902762 29281 raft_consensus.cc:504] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.903684 29281 leader_election.cc:288] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:13.904734 28604 tablet_bootstrap.cc:654] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.905090 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514b79467dc42bc89fdacf8e39a000d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:13.905366 28491 raft_consensus.cc:3037] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.907550 29424 raft_consensus.cc:504] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.908814 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f08aac9a7147ebaffda4262a57a86a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:13.909184 28491 raft_consensus.cc:2454] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:13.909664 28150 leader_election.cc:302] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:13.909962 29424 leader_election.cc:288] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:13.910215 28017 consensus_peers.cc:485] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 -> Peer a10ddca38c26462eb221d6026e88c6cc (127.27.50.253:44259): Couldn't send request to peer a10ddca38c26462eb221d6026e88c6cc. Status: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:12:13.910423 28150 leader_election.cc:341] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 3514b79467dc42bc89fdacf8e39a000d
I20240503 14:12:13.910737 28661 ts_tablet_manager.cc:1875] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.910880 29431 raft_consensus.cc:686] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
W20240503 14:12:13.911314 28427 leader_election.cc:341] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:13.911209 29431 consensus_queue.cc:237] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.911899 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f08aac9a7147ebaffda4262a57a86a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:13.912200 28215 raft_consensus.cc:3037] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.912523 29281 raft_consensus.cc:2781] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.912931 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f08aac9a7147ebaffda4262a57a86a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.915289 28215 raft_consensus.cc:2454] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:13.916055 28427 leader_election.cc:302] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:13.916182 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ab077988364bd687e54f90c3c265d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cd037e135904e2cab09cbf39fed0b13) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.916501 29317 raft_consensus.cc:2781] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.916715 29317 raft_consensus.cc:686] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:13.917112 29317 consensus_queue.cc:237] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.917579 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:13.918735 28614 tablet_bootstrap.cc:492] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:13.923540 28653 ts_tablet_manager.cc:1888] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.923775 28653 log.cc:1196] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/004d556f3ed44a7db1a52d205ecfd90b
I20240503 14:12:13.924165 28614 ts_tablet_manager.cc:1367] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.010s
I20240503 14:12:13.924350 28653 ts_tablet_manager.cc:1909] T 004d556f3ed44a7db1a52d205ecfd90b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:13.925267 28614 raft_consensus.cc:348] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.925559 28614 raft_consensus.cc:374] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.925514 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2795914729c948ed94f38a47c734d74d" } modify_external_catalogs: true
I20240503 14:12:13.925668 28614 raft_consensus.cc:729] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.925806 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2795914729c948ed94f38a47c734d74d" } modify_external_catalogs: true
I20240503 14:12:13.927492 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_68a22cfee7f644fcba9b9c3e34e6205c" } modify_external_catalogs: true
I20240503 14:12:13.927760 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_68a22cfee7f644fcba9b9c3e34e6205c" } modify_external_catalogs: true
I20240503 14:12:13.928596 28335 tablet_service.cc:1467] Processing CreateTablet for tablet d4b2b486f56f45cca8f015deaf61150a (DEFAULT_TABLE table=default.Table_ff836f0ad0b34e63bf9ee17c3f678da8 [id=5dc0316f7c044580a8864dbbd7de7d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.929211 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b2b486f56f45cca8f015deaf61150a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.929528 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 0b46cfed6c8e42a6acf4d1fb128a8656 (DEFAULT_TABLE table=default.Table_6b19c8158c64491dad1fdcddefb6b2b0 [id=0504eea8d2bf44c086b524d37d834f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.930054 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b46cfed6c8e42a6acf4d1fb128a8656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.930195 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 426) took 2309 ms. Trace:
I20240503 14:12:13.930366 28653 rpcz_store.cc:276] 0503 14:12:11.620397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.622511 (+ 2114us) service_pool.cc:225] Handling call
0503 14:12:13.680033 (+2057522us) ts_tablet_manager.cc:1120] Deleting tablet 004d556f3ed44a7db1a52d205ecfd90b
0503 14:12:13.680045 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.737422 (+ 57377us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.814556 (+ 77134us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.930144 (+115588us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2057155}
I20240503 14:12:13.930841 28653 tablet_replica.cc:330] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:13.931133 28653 raft_consensus.cc:2229] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.931339 28653 raft_consensus.cc:2258] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.929211 28326 tablet_service.cc:1467] Processing CreateTablet for tablet ac577af6460b423288a790e532b00f82 (DEFAULT_TABLE table=default.Table_c297c309fedc4148849b57c91f68d1b4 [id=88d060012eef400b97c4a62d199d194a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.931928 28614 consensus_queue.cc:260] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.933818 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 004d556f3ed44a7db1a52d205ecfd90b (table default.Table_5264956c580441a6a23c96d39a41ea71 [id=59887894cf0843df9cd01850035c1e64]) successfully deleted
I20240503 14:12:13.934535 28187 tablet_service.cc:1467] Processing CreateTablet for tablet d4b2b486f56f45cca8f015deaf61150a (DEFAULT_TABLE table=default.Table_ff836f0ad0b34e63bf9ee17c3f678da8 [id=5dc0316f7c044580a8864dbbd7de7d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.934979 28600 tablet_bootstrap.cc:654] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.935019 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b2b486f56f45cca8f015deaf61150a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.936100 28604 tablet_bootstrap.cc:492] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:13.936301 28604 ts_tablet_manager.cc:1367] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240503 14:12:13.936266 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 (DEFAULT_TABLE table=default.Table_808ffc4c24d0480b87dab400bd4aec7e [id=66e880194cf84854be7f2ddfbc09c1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.936779 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0d676fa1e74ab89922f7ba607fd2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.937330 28604 raft_consensus.cc:348] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.938087 28604 raft_consensus.cc:374] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.937996 28183 tablet_service.cc:1467] Processing CreateTablet for tablet ac577af6460b423288a790e532b00f82 (DEFAULT_TABLE table=default.Table_c297c309fedc4148849b57c91f68d1b4 [id=88d060012eef400b97c4a62d199d194a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.938189 28604 raft_consensus.cc:729] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.938498 28604 consensus_queue.cc:260] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.941028 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6b94092725fa4956a4c00e3685f9aeaf
I20240503 14:12:13.942035 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet aa3ce05b422a4166bfac468acb84e4cc
I20240503 14:12:13.942162 28604 ts_tablet_manager.cc:1398] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:13.942571 28604 tablet_bootstrap.cc:492] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:13.942656 28032 catalog_manager.cc:6554] ReplaceTablet: tablet aa3ce05b422a4166bfac468acb84e4cc deleted and replaced by tablet 64f226db2fdb47ba9472a3aa65b953c3
I20240503 14:12:13.943059 28661 ts_tablet_manager.cc:1888] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:13.943431 28661 log.cc:1196] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0b255d3e3dc64fb69c2ef6ed389db6a6
I20240503 14:12:13.944413 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3ce05b422a4166bfac468acb84e4cc with delete_type TABLET_DATA_DELETED (replaced by tablet 64f226db2fdb47ba9472a3aa65b953c3) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.944849 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac577af6460b423288a790e532b00f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.944885 28661 ts_tablet_manager.cc:1909] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:13.945354 28195 tablet_service.cc:1467] Processing CreateTablet for tablet ffcfc89a415e4c68b6a54318755fe0d7 (DEFAULT_TABLE table=default.Table_9150e78ff2d94d1dba8f6d1fd1727db2 [id=d41b289bddc44de2bb4b2a6145904d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.945797 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcfc89a415e4c68b6a54318755fe0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.946255 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac577af6460b423288a790e532b00f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.948817 28461 tablet_service.cc:1467] Processing CreateTablet for tablet c50ed206852541ec9413dbf358265301 (DEFAULT_TABLE table=default.Table_45877d24af954c5ebdd54673110911e6 [id=00a6a100a3d74675a67db0835b64bbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.950259 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50ed206852541ec9413dbf358265301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.951200 28458 tablet_service.cc:1467] Processing CreateTablet for tablet ffcfc89a415e4c68b6a54318755fe0d7 (DEFAULT_TABLE table=default.Table_9150e78ff2d94d1dba8f6d1fd1727db2 [id=d41b289bddc44de2bb4b2a6145904d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.953233 28469 tablet_service.cc:1467] Processing CreateTablet for tablet d4b2b486f56f45cca8f015deaf61150a (DEFAULT_TABLE table=default.Table_ff836f0ad0b34e63bf9ee17c3f678da8 [id=5dc0316f7c044580a8864dbbd7de7d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.949961 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 0b46cfed6c8e42a6acf4d1fb128a8656 (DEFAULT_TABLE table=default.Table_6b19c8158c64491dad1fdcddefb6b2b0 [id=0504eea8d2bf44c086b524d37d834f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.954119 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcfc89a415e4c68b6a54318755fe0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.932029 28653 ts_tablet_manager.cc:1875] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.955082 28465 tablet_service.cc:1467] Processing CreateTablet for tablet ac577af6460b423288a790e532b00f82 (DEFAULT_TABLE table=default.Table_c297c309fedc4148849b57c91f68d1b4 [id=88d060012eef400b97c4a62d199d194a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.955771 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac577af6460b423288a790e532b00f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.957001 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b2b486f56f45cca8f015deaf61150a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.957888 28603 raft_consensus.cc:2229] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:13.958175 28603 raft_consensus.cc:2258] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:13.958626 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b94092725fa4956a4c00e3685f9aeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.959932 28603 ts_tablet_manager.cc:1875] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:13.960280 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet c04707221b5241db8cd380ad5db24af5
I20240503 14:12:13.961750 28322 tablet_service.cc:1467] Processing CreateTablet for tablet f3c35f5b09e24cc69e2477d1a3946779 (DEFAULT_TABLE table=default.Table_d467def5cd244a9a9069718e6070c314 [id=46d40d7fe69e4b1097b56dfdfb82fcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.962312 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3c35f5b09e24cc69e2477d1a3946779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.963446 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3ce05b422a4166bfac468acb84e4cc with delete_type TABLET_DATA_DELETED (replaced by tablet 64f226db2fdb47ba9472a3aa65b953c3) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:13.964461 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 4fad4508a29d4822b81af6947b57e7ef
I20240503 14:12:13.969839 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b94092725fa4956a4c00e3685f9aeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.975639 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_07467425de964d5aa2a124eaba28268c" } new_table_name: "default.Table_e3c825867cf84b72bfb708fcc39f6acb" modify_external_catalogs: true
I20240503 14:12:13.977792 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_98e9ee5715624c0c804e368311ceabb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.978716 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98e9ee5715624c0c804e368311ceabb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.979239 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 0b46cfed6c8e42a6acf4d1fb128a8656 (DEFAULT_TABLE table=default.Table_6b19c8158c64491dad1fdcddefb6b2b0 [id=0504eea8d2bf44c086b524d37d834f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.980553 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_daa40e7d4485405eabf710a2ddc0a60e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.981333 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_daa40e7d4485405eabf710a2ddc0a60e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.981302 29281 raft_consensus.cc:686] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:13.982846 28192 tablet_service.cc:1467] Processing CreateTablet for tablet c50ed206852541ec9413dbf358265301 (DEFAULT_TABLE table=default.Table_45877d24af954c5ebdd54673110911e6 [id=00a6a100a3d74675a67db0835b64bbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.984532 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1307b27ba2a5418a950338dc5e1f5f3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:13.985268 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1307b27ba2a5418a950338dc5e1f5f3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:13.986244 29281 consensus_queue.cc:237] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:13.988029 28193 tablet_service.cc:1467] Processing CreateTablet for tablet f3c35f5b09e24cc69e2477d1a3946779 (DEFAULT_TABLE table=default.Table_d467def5cd244a9a9069718e6070c314 [id=46d40d7fe69e4b1097b56dfdfb82fcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.991760 28182 tablet_service.cc:1467] Processing CreateTablet for tablet f6061918660e4070a0c8339cb5009ff8 (DEFAULT_TABLE table=default.Table_9f5a767355f24f1b928c442018fe05c3 [id=0cc40a179e564e95bbeea488c5640c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.992635 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 84c943a93a814633b7f71a4cb55b5ee3 (DEFAULT_TABLE table=default.Table_7667eac51a7246a595705267a0d758a2 [id=8c953739934641d4a030a7ce771b1b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.996517 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 58fe166ca0634e5eb1ae4cba526f494c (DEFAULT_TABLE table=default.Table_52c51767da7d4d86b088b1bb16ce9623 [id=1a8e59318503488d8bf4552ed9eb8364]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.996930 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 10876f4512794e6aa72bda3c1da8b2d0 (DEFAULT_TABLE table=default.Table_c8a1b23185cf4c51ac94cb268e9e2222 [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.950681 28321 tablet_service.cc:1467] Processing CreateTablet for tablet c50ed206852541ec9413dbf358265301 (DEFAULT_TABLE table=default.Table_45877d24af954c5ebdd54673110911e6 [id=00a6a100a3d74675a67db0835b64bbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.999420 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fad4508a29d4822b81af6947b57e7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:13.999464 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b46cfed6c8e42a6acf4d1fb128a8656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.000162 28189 tablet_service.cc:1467] Processing CreateTablet for tablet be238ee1d82548c9a1940e2a72e395c7 (DEFAULT_TABLE table=default.Table_9360be1891a744a48835f3976afd1059 [id=2286e92e6cfe450894f2d9ca699f91f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.958376 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 (DEFAULT_TABLE table=default.Table_808ffc4c24d0480b87dab400bd4aec7e [id=66e880194cf84854be7f2ddfbc09c1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.003641 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 306adb5eecc14e099626f91fb4a3d04c (DEFAULT_TABLE table=default.Table_8f9db6955d4e47c18e50ceefd3d230f8 [id=e26e6bda9d874c9983342cec4322657d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.003602 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9f158ace156f44beacdd67afbd7eafc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.004050 28180 tablet_service.cc:1467] Processing CreateTablet for tablet e124feb6bfc8462d8d8f6f222215634c (DEFAULT_TABLE table=default.Table_0ac41173b58543dcaa1508ac89ee3a4e [id=ac9fd80aef0c434dab4a5729fdda282b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.004359 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f158ace156f44beacdd67afbd7eafc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.005640 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 98d33cc314ac4e53af8734bd752d2d27 (DEFAULT_TABLE table=default.Table_afbdf248811d4722b5fa2c30b6f9780f [id=c978e5b427974b40bb5d3ce16f950ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.005906 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:13.946827 28334 tablet_service.cc:1467] Processing CreateTablet for tablet ffcfc89a415e4c68b6a54318755fe0d7 (DEFAULT_TABLE table=default.Table_9150e78ff2d94d1dba8f6d1fd1727db2 [id=d41b289bddc44de2bb4b2a6145904d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.007565 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4852bd62507141ed994fa106a0927878" } new_table_name: "default.Table_69dd549e83174c59bc6c2142ccb73565" modify_external_catalogs: true
I20240503 14:12:14.009354 28178 tablet_service.cc:1467] Processing CreateTablet for tablet dee5a230ef584077ba9f4a27c878ef4d (DEFAULT_TABLE table=default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.011021 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 84c943a93a814633b7f71a4cb55b5ee3 (DEFAULT_TABLE table=default.Table_7667eac51a7246a595705267a0d758a2 [id=8c953739934641d4a030a7ce771b1b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:13.996371 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7aba6cf95a4e4459aa96ea157115e3d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.011799 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 10876f4512794e6aa72bda3c1da8b2d0 (DEFAULT_TABLE table=default.Table_c8a1b23185cf4c51ac94cb268e9e2222 [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.012066 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aba6cf95a4e4459aa96ea157115e3d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.012329 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fad4508a29d4822b81af6947b57e7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.012671 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c943a93a814633b7f71a4cb55b5ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.013402 28336 tablet_service.cc:1467] Processing CreateTablet for tablet f6061918660e4070a0c8339cb5009ff8 (DEFAULT_TABLE table=default.Table_9f5a767355f24f1b928c442018fe05c3 [id=0cc40a179e564e95bbeea488c5640c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.014086 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 306adb5eecc14e099626f91fb4a3d04c (DEFAULT_TABLE table=default.Table_8f9db6955d4e47c18e50ceefd3d230f8 [id=e26e6bda9d874c9983342cec4322657d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.014674 28331 tablet_service.cc:1467] Processing CreateTablet for tablet e124feb6bfc8462d8d8f6f222215634c (DEFAULT_TABLE table=default.Table_0ac41173b58543dcaa1508ac89ee3a4e [id=ac9fd80aef0c434dab4a5729fdda282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.015239 28327 tablet_service.cc:1467] Processing CreateTablet for tablet dee5a230ef584077ba9f4a27c878ef4d (DEFAULT_TABLE table=default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.015714 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet c04707221b5241db8cd380ad5db24af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:14.015779 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7218cefb13ea4983b2111ed5fd0c997b consensus state has no leader
I20240503 14:12:14.016136 28323 tablet_service.cc:1467] Processing CreateTablet for tablet be238ee1d82548c9a1940e2a72e395c7 (DEFAULT_TABLE table=default.Table_9360be1891a744a48835f3976afd1059 [id=2286e92e6cfe450894f2d9ca699f91f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.015908 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 84c943a93a814633b7f71a4cb55b5ee3 (DEFAULT_TABLE table=default.Table_7667eac51a7246a595705267a0d758a2 [id=8c953739934641d4a030a7ce771b1b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.019783 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 58fe166ca0634e5eb1ae4cba526f494c (DEFAULT_TABLE table=default.Table_52c51767da7d4d86b088b1bb16ce9623 [id=1a8e59318503488d8bf4552ed9eb8364]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.022292 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet c04707221b5241db8cd380ad5db24af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.025333 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3ce05b422a4166bfac468acb84e4cc with delete_type TABLET_DATA_DELETED (replaced by tablet 64f226db2fdb47ba9472a3aa65b953c3) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.025668 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50ed206852541ec9413dbf358265301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.025818 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 466) took 1470 ms. Trace:
I20240503 14:12:14.025944 28661 rpcz_store.cc:276] 0503 14:12:12.555066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.555130 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:13.825985 (+1270855us) ts_tablet_manager.cc:1120] Deleting tablet 0b255d3e3dc64fb69c2ef6ed389db6a6
0503 14:12:13.825997 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:13.914550 (+ 88553us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:13.942878 (+ 28328us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:14.025769 (+ 82891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269683}
I20240503 14:12:14.026400 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 469) took 1407 ms. Trace:
W20240503 14:12:14.026582 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d832cd30f2414303818d34a0cdf47e60 consensus state has no leader
I20240503 14:12:14.026517 28661 rpcz_store.cc:276] 0503 14:12:12.618982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.619163 (+ 181us) service_pool.cc:225] Handling call
0503 14:12:14.026227 (+1407064us) ts_tablet_manager.cc:1120] Deleting tablet b476e87faea1415d95ae014a93fa22af
0503 14:12:14.026239 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.026364 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406435}
I20240503 14:12:14.029384 28614 ts_tablet_manager.cc:1398] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.094s user 0.003s sys 0.000s
I20240503 14:12:14.029551 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 98d33cc314ac4e53af8734bd752d2d27 (DEFAULT_TABLE table=default.Table_afbdf248811d4722b5fa2c30b6f9780f [id=c978e5b427974b40bb5d3ce16f950ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.032044 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d33cc314ac4e53af8734bd752d2d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.036231 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0d676fa1e74ab89922f7ba607fd2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.991727 28600 tablet_bootstrap.cc:492] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:14.038082 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10876f4512794e6aa72bda3c1da8b2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.038975 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c943a93a814633b7f71a4cb55b5ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.040766 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3c35f5b09e24cc69e2477d1a3946779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.040419 28600 ts_tablet_manager.cc:1367] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.159s user 0.003s sys 0.020s
I20240503 14:12:14.042186 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d33cc314ac4e53af8734bd752d2d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.045281 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b46cfed6c8e42a6acf4d1fb128a8656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.047533 28614 tablet_bootstrap.cc:492] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:14.049480 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b94092725fa4956a4c00e3685f9aeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.050796 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcfc89a415e4c68b6a54318755fe0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.050797 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 98d33cc314ac4e53af8734bd752d2d27 (DEFAULT_TABLE table=default.Table_afbdf248811d4722b5fa2c30b6f9780f [id=c978e5b427974b40bb5d3ce16f950ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.052346 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6061918660e4070a0c8339cb5009ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.054929 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_730611e6ad48480eb3f4498f3e79725b" } modify_external_catalogs: true
I20240503 14:12:14.055212 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_730611e6ad48480eb3f4498f3e79725b" } modify_external_catalogs: true
I20240503 14:12:14.057548 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be238ee1d82548c9a1940e2a72e395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.058027 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b80186a1b50741aca2ee3ff55a4ef6be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.058880 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b80186a1b50741aca2ee3ff55a4ef6be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.065176 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee5a230ef584077ba9f4a27c878ef4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.066332 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d5c70a4285c24a50b556481cf3793c19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.067204 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5c70a4285c24a50b556481cf3793c19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.067483 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fe166ca0634e5eb1ae4cba526f494c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.072351 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6061918660e4070a0c8339cb5009ff8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.080798 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7218cefb13ea4983b2111ed5fd0c997b (table default.Table_e3c825867cf84b72bfb708fcc39f6acb [id=3a24797cc53b439da44cc77025fa9f92], current schema version=8) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7218cefb13ea4983b2111ed5fd0c997b consensus state has no leader
I20240503 14:12:14.081409 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8f505fa519514278ae7489b9ff7bbaf2" } modify_external_catalogs: true
I20240503 14:12:14.081735 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8f505fa519514278ae7489b9ff7bbaf2" } modify_external_catalogs: true
I20240503 14:12:14.082578 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b1ae9903413045af9a9fa56d22ac1552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.083395 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1ae9903413045af9a9fa56d22ac1552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.084416 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_96e46f964a63484bb1f66f90e65c72db" } new_table_name: "default.Table_1de6580ff46348ffa25a2704e01ea777" modify_external_catalogs: true
I20240503 14:12:14.086344 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_69dd549e83174c59bc6c2142ccb73565" } new_table_name: "default.Table_831065d014d241bfbb471ac2cfd336bd" modify_external_catalogs: true
I20240503 14:12:14.089565 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_49cdd819727f4ad381e0aae8fba8d201" } new_table_name: "default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb" modify_external_catalogs: true
I20240503 14:12:14.091593 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 10876f4512794e6aa72bda3c1da8b2d0 (DEFAULT_TABLE table=default.Table_c8a1b23185cf4c51ac94cb268e9e2222 [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.093636 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 (DEFAULT_TABLE table=default.Table_808ffc4c24d0480b87dab400bd4aec7e [id=66e880194cf84854be7f2ddfbc09c1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.094631 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 58fe166ca0634e5eb1ae4cba526f494c (DEFAULT_TABLE table=default.Table_52c51767da7d4d86b088b1bb16ce9623 [id=1a8e59318503488d8bf4552ed9eb8364]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.095096 28464 tablet_service.cc:1467] Processing CreateTablet for tablet f3c35f5b09e24cc69e2477d1a3946779 (DEFAULT_TABLE table=default.Table_d467def5cd244a9a9069718e6070c314 [id=46d40d7fe69e4b1097b56dfdfb82fcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.102463 28462 tablet_service.cc:1467] Processing CreateTablet for tablet f6061918660e4070a0c8339cb5009ff8 (DEFAULT_TABLE table=default.Table_9f5a767355f24f1b928c442018fe05c3 [id=0cc40a179e564e95bbeea488c5640c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.103085 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c943a93a814633b7f71a4cb55b5ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.112319 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0d676fa1e74ab89922f7ba607fd2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.118538 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10876f4512794e6aa72bda3c1da8b2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.125221 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e124feb6bfc8462d8d8f6f222215634c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.125893 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fe166ca0634e5eb1ae4cba526f494c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.126417 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 306adb5eecc14e099626f91fb4a3d04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.126897 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3c35f5b09e24cc69e2477d1a3946779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.127951 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6061918660e4070a0c8339cb5009ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.129047 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d33cc314ac4e53af8734bd752d2d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.129889 28604 tablet_bootstrap.cc:654] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.134971 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 (table default.Table_068cea3b090241c7980d0edb57c05aec [id=80b934d1638444ebb855ebacf917c23e]) successfully deleted
W20240503 14:12:14.135401 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b476e87faea1415d95ae014a93fa22af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b476e87faea1415d95ae014a93fa22af already in progress: creating tablet
I20240503 14:12:14.136703 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.138337 28614 tablet_bootstrap.cc:654] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.142446 29431 raft_consensus.cc:482] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.142729 29431 raft_consensus.cc:504] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.133520 28459 tablet_service.cc:1467] Processing CreateTablet for tablet e124feb6bfc8462d8d8f6f222215634c (DEFAULT_TABLE table=default.Table_0ac41173b58543dcaa1508ac89ee3a4e [id=ac9fd80aef0c434dab4a5729fdda282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.144635 29431 leader_election.cc:288] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:14.147189 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fad4508a29d4822b81af6947b57e7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.147432 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet c04707221b5241db8cd380ad5db24af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.147583 28466 tablet_service.cc:1467] Processing CreateTablet for tablet dee5a230ef584077ba9f4a27c878ef4d (DEFAULT_TABLE table=default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.147985 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 306adb5eecc14e099626f91fb4a3d04c (DEFAULT_TABLE table=default.Table_8f9db6955d4e47c18e50ceefd3d230f8 [id=e26e6bda9d874c9983342cec4322657d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.149906 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 470) took 1519 ms. Trace:
I20240503 14:12:14.146981 28468 tablet_service.cc:1467] Processing CreateTablet for tablet be238ee1d82548c9a1940e2a72e395c7 (DEFAULT_TABLE table=default.Table_9360be1891a744a48835f3976afd1059 [id=2286e92e6cfe450894f2d9ca699f91f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.159004 28661 rpcz_store.cc:276] 0503 14:12:12.619111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.619311 (+ 200us) service_pool.cc:225] Handling call
0503 14:12:14.026753 (+1407442us) ts_tablet_manager.cc:1120] Deleting tablet d4c8a97076a44f9eaa3a15026f90a304
0503 14:12:14.026763 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.138724 (+111961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406596}
I20240503 14:12:14.160498 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 471) took 1441 ms. Trace:
I20240503 14:12:14.160854 28661 rpcz_store.cc:276] 0503 14:12:12.718849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.719332 (+ 483us) service_pool.cc:225] Handling call
0503 14:12:14.159990 (+1440658us) ts_tablet_manager.cc:1120] Deleting tablet 7b6c998e518541248a87e4c1551ec080
0503 14:12:14.160042 (+ 52us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.160353 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1440331}
I20240503 14:12:14.161716 28661 tablet_replica.cc:330] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:14.166421 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e124feb6bfc8462d8d8f6f222215634c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.168634 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee5a230ef584077ba9f4a27c878ef4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.169946 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 306adb5eecc14e099626f91fb4a3d04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.170472 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be238ee1d82548c9a1940e2a72e395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.172451 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044f34588d045ce898435b61a90503f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:14.205374 28603 ts_tablet_manager.cc:1888] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.205644 28603 log.cc:1196] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/63e977f7837e49fb93ef0fd18b619792
W20240503 14:12:14.178344 28427 consensus_peers.cc:485] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 651478ef632144079f93f3b50bff4611. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:14.212291 28661 raft_consensus.cc:2229] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.212658 28661 raft_consensus.cc:2258] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.213493 28661 ts_tablet_manager.cc:1875] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:14.216739 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d4c8a97076a44f9eaa3a15026f90a304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c8a97076a44f9eaa3a15026f90a304 already in progress: creating tablet
W20240503 14:12:14.221390 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7b6c998e518541248a87e4c1551ec080 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b6c998e518541248a87e4c1551ec080 already in progress: creating tablet
I20240503 14:12:14.240244 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6ec7fc5bf73b4e3aa4da80a76c3b2611"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.240980 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ec7fc5bf73b4e3aa4da80a76c3b2611 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.244870 28603 ts_tablet_manager.cc:1909] T 63e977f7837e49fb93ef0fd18b619792 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.245781 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 63e977f7837e49fb93ef0fd18b619792 (table default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4 [id=dcdb4c8fe4774779b0303db262dbfcc8]) successfully deleted
W20240503 14:12:14.250504 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 920b41e3b21347b393497de4d4e87469 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920b41e3b21347b393497de4d4e87469 already in progress: creating tablet
W20240503 14:12:14.255086 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9b21cd1262d94d1c994b29c44f86cb5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b21cd1262d94d1c994b29c44f86cb5e already in progress: creating tablet
I20240503 14:12:14.256234 28603 tablet_replica.cc:330] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.256505 28603 raft_consensus.cc:2229] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.261134 28604 tablet_bootstrap.cc:492] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:14.261350 28604 ts_tablet_manager.cc:1367] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.319s user 0.000s sys 0.006s
I20240503 14:12:14.262882 28604 raft_consensus.cc:348] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.263160 28604 raft_consensus.cc:374] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.264521 28604 raft_consensus.cc:729] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.264932 28604 consensus_queue.cc:260] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.264041 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1efd634aa4c0450283b143ce587da86e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.268759 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1efd634aa4c0450283b143ce587da86e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.271220 28604 ts_tablet_manager.cc:1398] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
W20240503 14:12:14.279245 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 150733us
I20240503 14:12:14.280161 28603 raft_consensus.cc:2258] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.283664 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10876f4512794e6aa72bda3c1da8b2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.284163 28603 ts_tablet_manager.cc:1875] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:14.285768 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50ed206852541ec9413dbf358265301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.286449 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fe166ca0634e5eb1ae4cba526f494c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.288507 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee5a230ef584077ba9f4a27c878ef4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.288534 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044f34588d045ce898435b61a90503f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:14.289335 28217 raft_consensus.cc:2454] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:14.292800 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be238ee1d82548c9a1940e2a72e395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.300503 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e124feb6bfc8462d8d8f6f222215634c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.302402 28604 tablet_bootstrap.cc:492] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:14.304258 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e977f7837e49fb93ef0fd18b619792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.305189 28604 tablet_bootstrap.cc:654] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.306044 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 306adb5eecc14e099626f91fb4a3d04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.310308 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044f34588d045ce898435b61a90503f with delete_type TABLET_DATA_DELETED (replaced by tablet daf5d153ab5142c0902cc6e96629f412) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:14.311889 28427 leader_election.cc:341] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:14.318866 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc31b2e907b04ea19a6f9f7fd8f57843 consensus state has no leader
I20240503 14:12:14.326052 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ea4ed2f332dc46aaa1f6e7025a1a1c3c" } modify_external_catalogs: true
I20240503 14:12:14.326354 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ea4ed2f332dc46aaa1f6e7025a1a1c3c" } modify_external_catalogs: true
W20240503 14:12:14.335992 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d832cd30f2414303818d34a0cdf47e60 consensus state has no leader
I20240503 14:12:14.336496 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1307b27ba2a5418a950338dc5e1f5f3a" } modify_external_catalogs: true
I20240503 14:12:14.336766 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1307b27ba2a5418a950338dc5e1f5f3a" } modify_external_catalogs: true
I20240503 14:12:14.338965 28427 leader_election.cc:302] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:14.339347 29431 raft_consensus.cc:2781] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:14.340023 29431 raft_consensus.cc:482] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:14.340210 29431 raft_consensus.cc:3037] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.341169 29429 consensus_queue.cc:1035] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:14.342260 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.344055 28661 ts_tablet_manager.cc:1888] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.349330 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d8cb0703edfb4e96ba081bd6f299d867" } modify_external_catalogs: true
I20240503 14:12:14.349615 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d8cb0703edfb4e96ba081bd6f299d867" } modify_external_catalogs: true
I20240503 14:12:14.351176 29480 raft_consensus.cc:482] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.351449 28661 log.cc:1196] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/265dc5f2630e453ea71abddb1a9753f3
I20240503 14:12:14.352135 28661 ts_tablet_manager.cc:1909] T 265dc5f2630e453ea71abddb1a9753f3 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:14.353127 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 472) took 1631 ms. Trace:
I20240503 14:12:14.353303 28661 rpcz_store.cc:276] 0503 14:12:12.721356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.721413 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:14.161557 (+1440144us) ts_tablet_manager.cc:1120] Deleting tablet 265dc5f2630e453ea71abddb1a9753f3
0503 14:12:14.161578 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.312500 (+150922us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:14.343917 (+ 31417us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:14.353054 (+ 9137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439865}
I20240503 14:12:14.353991 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 474) took 1630 ms. Trace:
I20240503 14:12:14.354233 28661 rpcz_store.cc:276] 0503 14:12:12.723021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.723077 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:14.353749 (+1630672us) ts_tablet_manager.cc:1120] Deleting tablet ce4846d662a64d51b16fd5447c371e9a
0503 14:12:14.353760 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.353903 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1630420}
I20240503 14:12:14.354804 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 493) took 1165 ms. Trace:
I20240503 14:12:14.354982 28661 rpcz_store.cc:276] 0503 14:12:13.189356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.189467 (+ 111us) service_pool.cc:225] Handling call
0503 14:12:14.354577 (+1165110us) ts_tablet_manager.cc:1120] Deleting tablet 320c526e845e4dc78da345bcb095918f
0503 14:12:14.354587 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.354713 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164794}
I20240503 14:12:14.355496 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 494) took 1165 ms. Trace:
I20240503 14:12:14.355681 28661 rpcz_store.cc:276] 0503 14:12:13.190291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.190350 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:14.355293 (+1164943us) ts_tablet_manager.cc:1120] Deleting tablet 4084bd4fbf604042bb33ff03c3f8cc19
0503 14:12:14.355302 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.355409 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164273}
I20240503 14:12:14.356324 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 496) took 1153 ms. Trace:
I20240503 14:12:14.356504 28661 rpcz_store.cc:276] 0503 14:12:13.202489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.205738 (+ 3249us) service_pool.cc:225] Handling call
0503 14:12:14.356026 (+1150288us) ts_tablet_manager.cc:1120] Deleting tablet ca6627f0d9a544bb8176bea6c1af5ab4
0503 14:12:14.356035 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.356193 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149585}
I20240503 14:12:14.357023 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 497) took 1129 ms. Trace:
W20240503 14:12:14.357314 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7261088abbc641db8d737a48bbac25bd consensus state has no leader
I20240503 14:12:14.357921 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6b41ac2dcacf495bbb627aa567c42764"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.358911 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b41ac2dcacf495bbb627aa567c42764 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.360968 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b86f741291e64aef9908c5e036e4cce8" } new_table_name: "default.Table_94d6ea5674494c8a960eaee9c10ded44" modify_external_catalogs: true
I20240503 14:12:14.351401 29480 raft_consensus.cc:504] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.362443 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4c5648ddc84407a8d8b44f2d63dcee (DEFAULT_TABLE table=default.Table_0ddd72e685204e87a5edb2761dda0b0c [id=8e6caae526a44e5f828b9ddfebf6d426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.362706 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 73b9c0391d7f457ebdd421571ed64cf1 (DEFAULT_TABLE table=default.Table_d4cc46240b5f4f88bbf7c96f7a78d0c3 [id=9037e3697ffc45bf892d529d3096e802]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.362955 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 09f1f5216cc940888803ce8f6acd2a47 (DEFAULT_TABLE table=default.Table_9f6a864cf34b498f87ccf5afca7d29e8 [id=b44143b111d6462793be1f5aa91fd6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.363193 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73b9c0391d7f457ebdd421571ed64cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.364148 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a1bf5785458b47d0a18f3f3088449ea3" } new_table_name: "default.Table_ddd5f026bb204de99edef3cc54bd8347" modify_external_catalogs: true
I20240503 14:12:14.366529 29480 leader_election.cc:288] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:14.372735 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f1f5216cc940888803ce8f6acd2a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.374425 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f41bb9c5d40b0913264864ecc3528" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:14.375475 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4c5648ddc84407a8d8b44f2d63dcee (DEFAULT_TABLE table=default.Table_0ddd72e685204e87a5edb2761dda0b0c [id=8e6caae526a44e5f828b9ddfebf6d426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.380549 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4c5648ddc84407a8d8b44f2d63dcee. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.381021 28150 leader_election.cc:341] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:14.386104 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4c5648ddc84407a8d8b44f2d63dcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.382181 28195 tablet_service.cc:1467] Processing CreateTablet for tablet db88ca39eba44740994dd05b526e8506 (DEFAULT_TABLE table=default.Table_d0e52f75ad7f415bac0142d93ff4aa78 [id=47c9250b29ba4d78a9c18d29ad5b899a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.387835 28600 raft_consensus.cc:348] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.389598 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 73b9c0391d7f457ebdd421571ed64cf1 (DEFAULT_TABLE table=default.Table_d4cc46240b5f4f88bbf7c96f7a78d0c3 [id=9037e3697ffc45bf892d529d3096e802]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.390038 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 09f1f5216cc940888803ce8f6acd2a47 (DEFAULT_TABLE table=default.Table_9f6a864cf34b498f87ccf5afca7d29e8 [id=b44143b111d6462793be1f5aa91fd6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.390125 28600 raft_consensus.cc:374] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.390282 28600 raft_consensus.cc:729] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.390318 28325 tablet_service.cc:1467] Processing CreateTablet for tablet d3943c8358a3484d943bad0c906a9425 (DEFAULT_TABLE table=default.Table_738df9de21164980b362ce19f71f3d9b [id=daa11e25d86b4147bfafb721ebd6654c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.390949 28614 tablet_bootstrap.cc:492] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:14.391290 28600 consensus_queue.cc:260] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.391399 28661 rpcz_store.cc:276] 0503 14:12:13.227269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.227480 (+ 211us) service_pool.cc:225] Handling call
0503 14:12:14.356836 (+1129356us) ts_tablet_manager.cc:1120] Deleting tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df
0503 14:12:14.356844 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.356938 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1128970}
I20240503 14:12:14.392026 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 927d4359637f4ec189a5143e8201c7f1 (DEFAULT_TABLE table=default.Table_eeca27b59dd543faa1ea8c60944ddadf [id=6266e38bb09d45fdbee6634a0e57335c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.392140 28600 ts_tablet_manager.cc:1398] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.351s user 0.003s sys 0.000s
I20240503 14:12:14.365886 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f41bb9c5d40b0913264864ecc3528" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:14.392908 28661 tablet_replica.cc:330] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:14.393043 28491 raft_consensus.cc:2454] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:14.393173 28661 raft_consensus.cc:2229] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.367244 28469 tablet_service.cc:1467] Processing CreateTablet for tablet db88ca39eba44740994dd05b526e8506 (DEFAULT_TABLE table=default.Table_d0e52f75ad7f415bac0142d93ff4aa78 [id=47c9250b29ba4d78a9c18d29ad5b899a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.393512 28661 raft_consensus.cc:2258] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.393718 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db88ca39eba44740994dd05b526e8506. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.394090 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c2307b8e64c2481d5bd47e9189fea (DEFAULT_TABLE table=default.Table_7e0f5faa00554a3b9cdaff8e346d048a [id=d34e9f2f945748988e15f9ae5a981b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.394456 28324 tablet_service.cc:1467] Processing CreateTablet for tablet db88ca39eba44740994dd05b526e8506 (DEFAULT_TABLE table=default.Table_d0e52f75ad7f415bac0142d93ff4aa78 [id=47c9250b29ba4d78a9c18d29ad5b899a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.395587 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73b9c0391d7f457ebdd421571ed64cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.368922 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4c5648ddc84407a8d8b44f2d63dcee (DEFAULT_TABLE table=default.Table_0ddd72e685204e87a5edb2761dda0b0c [id=8e6caae526a44e5f828b9ddfebf6d426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.369489 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 73b9c0391d7f457ebdd421571ed64cf1 (DEFAULT_TABLE table=default.Table_d4cc46240b5f4f88bbf7c96f7a78d0c3 [id=9037e3697ffc45bf892d529d3096e802]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.371407 29431 raft_consensus.cc:504] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.372949 28604 tablet_bootstrap.cc:492] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:14.396742 28604 ts_tablet_manager.cc:1367] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.094s user 0.010s sys 0.013s
I20240503 14:12:14.388792 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 927d4359637f4ec189a5143e8201c7f1 (DEFAULT_TABLE table=default.Table_eeca27b59dd543faa1ea8c60944ddadf [id=6266e38bb09d45fdbee6634a0e57335c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.389307 28460 tablet_service.cc:1467] Processing CreateTablet for tablet d3943c8358a3484d943bad0c906a9425 (DEFAULT_TABLE table=default.Table_738df9de21164980b362ce19f71f3d9b [id=daa11e25d86b4147bfafb721ebd6654c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.390888 28473 tablet_service.cc:1467] Processing CreateTablet for tablet ed177731c989482ebda1694dabcb8fb8 (DEFAULT_TABLE table=default.Table_3f6ca1c2e7314e7ebb2235561072ed1c [id=b396f957941242c7a09ef0bd312ff25c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.397871 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed177731c989482ebda1694dabcb8fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.398253 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927d4359637f4ec189a5143e8201c7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.398715 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3943c8358a3484d943bad0c906a9425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.399904 29431 leader_election.cc:288] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:14.399899 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 64f226db2fdb47ba9472a3aa65b953c3 (DEFAULT_TABLE table=default.Table_4e0a1dcdeddb46cfac04de76adfc27e5 [id=c53d59b2f4634fd7b3c26f749d9acc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.400935 28327 tablet_service.cc:1467] Processing CreateTablet for tablet ed177731c989482ebda1694dabcb8fb8 (DEFAULT_TABLE table=default.Table_3f6ca1c2e7314e7ebb2235561072ed1c [id=b396f957941242c7a09ef0bd312ff25c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.401382 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c2307b8e64c2481d5bd47e9189fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.401579 28600 tablet_bootstrap.cc:492] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:14.403107 28614 ts_tablet_manager.cc:1367] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.361s user 0.005s sys 0.011s
I20240503 14:12:14.402433 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 265dc5f2630e453ea71abddb1a9753f3 (table default.Table_cba5fd0a90c8433980631ba7e2048d8b [id=3368a0430dc94c4db0ae6260f5a0acf0]) successfully deleted
I20240503 14:12:14.405872 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed177731c989482ebda1694dabcb8fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.407755 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8c7d57fc4486457e9848daad79f04356
I20240503 14:12:14.409179 28197 tablet_service.cc:1467] Processing CreateTablet for tablet d3943c8358a3484d943bad0c906a9425 (DEFAULT_TABLE table=default.Table_738df9de21164980b362ce19f71f3d9b [id=daa11e25d86b4147bfafb721ebd6654c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.409857 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ce4846d662a64d51b16fd5447c371e9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce4846d662a64d51b16fd5447c371e9a already in progress: creating tablet
W20240503 14:12:14.410282 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 320c526e845e4dc78da345bcb095918f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 320c526e845e4dc78da345bcb095918f already in progress: creating tablet
I20240503 14:12:14.412039 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db88ca39eba44740994dd05b526e8506. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.412328 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f1f5216cc940888803ce8f6acd2a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.413395 28150 leader_election.cc:302] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:14.413878 29480 raft_consensus.cc:2781] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:14.414992 28653 ts_tablet_manager.cc:1888] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.416246 29480 raft_consensus.cc:482] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:14.416446 29480 raft_consensus.cc:3037] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:14.417344 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 4084bd4fbf604042bb33ff03c3f8cc19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4084bd4fbf604042bb33ff03c3f8cc19 already in progress: creating tablet
I20240503 14:12:14.417543 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_82dca5d2c206486786a35bb34369e2d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.417799 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ca6627f0d9a544bb8176bea6c1af5ab4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca6627f0d9a544bb8176bea6c1af5ab4 already in progress: creating tablet
W20240503 14:12:14.418817 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82dca5d2c206486786a35bb34369e2d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.420194 28328 tablet_service.cc:1467] Processing CreateTablet for tablet b0993be031cc4792a9eadba9ce41d024 (DEFAULT_TABLE table=default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.420606 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927d4359637f4ec189a5143e8201c7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.420904 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3943c8358a3484d943bad0c906a9425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.422051 28614 raft_consensus.cc:348] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.422338 28614 raft_consensus.cc:374] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.423609 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3943c8358a3484d943bad0c906a9425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.423177 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 927d4359637f4ec189a5143e8201c7f1 (DEFAULT_TABLE table=default.Table_eeca27b59dd543faa1ea8c60944ddadf [id=6266e38bb09d45fdbee6634a0e57335c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.427652 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c2307b8e64c2481d5bd47e9189fea (DEFAULT_TABLE table=default.Table_7e0f5faa00554a3b9cdaff8e346d048a [id=d34e9f2f945748988e15f9ae5a981b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.428568 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 64f226db2fdb47ba9472a3aa65b953c3 (DEFAULT_TABLE table=default.Table_4e0a1dcdeddb46cfac04de76adfc27e5 [id=c53d59b2f4634fd7b3c26f749d9acc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.429008 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0993be031cc4792a9eadba9ce41d024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.435134 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db88ca39eba44740994dd05b526e8506. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.435518 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb85d6e65b04937b3f89f8dcccd6df2 (DEFAULT_TABLE table=default.Table_d4a44441aa4647449f447922587e6ffd [id=3311b8b2dcf34b8583210f2e08eaca36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.436220 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_62b7129591554425bc6c23af414fbffb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.439937 28326 tablet_service.cc:1467] Processing CreateTablet for tablet e21a5dafa82f4660927d96c779a23478 (DEFAULT_TABLE table=default.Table_cc8ca3c970a949fab24298af4538c323 [id=ee1e18b4b41042199f351112b57335b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.440512 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7d57fc4486457e9848daad79f04356 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.441015 28603 ts_tablet_manager.cc:1888] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.448449 28603 log.cc:1196] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0bd5234791524eddb7edd493e39fb993
I20240503 14:12:14.450423 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb85d6e65b04937b3f89f8dcccd6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.453882 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9d878dfb144e5db914f8619c13ce8c (DEFAULT_TABLE table=default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.454480 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044f34588d045ce898435b61a90503f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:14.455010 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21a5dafa82f4660927d96c779a23478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.458269 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9d878dfb144e5db914f8619c13ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.458729 28661 ts_tablet_manager.cc:1875] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:14.458949 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4c5648ddc84407a8d8b44f2d63dcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.461025 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet c9c0540960ac45aabeaf13f888caf238
I20240503 14:12:14.459870 28604 raft_consensus.cc:348] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.463523 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb85d6e65b04937b3f89f8dcccd6df2 (DEFAULT_TABLE table=default.Table_d4a44441aa4647449f447922587e6ffd [id=3311b8b2dcf34b8583210f2e08eaca36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.464468 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7d57fc4486457e9848daad79f04356 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.465224 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 920be6d87c4440d0bf7453c86cfe0b5c
I20240503 14:12:14.465687 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd037e135904e2cab09cbf39fed0b13 (DEFAULT_TABLE table=default.Table_d121667544004d9e83e4daf3f58f7473 [id=7877e3c7f4b14fcebfd7bec2e15b6240]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.466113 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8097ba5606cf42d68448833de36d85c7 consensus state has no leader
W20240503 14:12:14.466378 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3d1314841574fc8bcf3ccb16755e421 consensus state has no leader
I20240503 14:12:14.466589 28027 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet ecf2bc5644874198b43e8071d2bc0071
I20240503 14:12:14.466782 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49
W20240503 14:12:14.467048 28427 leader_election.cc:341] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:14.468492 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73b9c0391d7f457ebdd421571ed64cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.468963 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 deleted and replaced by tablet 752b369f24224f2486834814a9d08495
I20240503 14:12:14.469377 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 77f0081cc2cb48aaa368741109032fd1
I20240503 14:12:14.469421 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 09f1f5216cc940888803ce8f6acd2a47 (DEFAULT_TABLE table=default.Table_9f6a864cf34b498f87ccf5afca7d29e8 [id=b44143b111d6462793be1f5aa91fd6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.470219 28457 tablet_service.cc:1467] Processing CreateTablet for tablet b0993be031cc4792a9eadba9ce41d024 (DEFAULT_TABLE table=default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.470585 28604 raft_consensus.cc:374] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.470728 28604 raft_consensus.cc:729] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.469941 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_15571f0ea45241568811001dfead7fff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.470085 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd037e135904e2cab09cbf39fed0b13 (DEFAULT_TABLE table=default.Table_d121667544004d9e83e4daf3f58f7473 [id=7877e3c7f4b14fcebfd7bec2e15b6240]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.472051 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15571f0ea45241568811001dfead7fff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.472937 28604 consensus_queue.cc:260] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.473976 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 752b369f24224f2486834814a9d08495) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.474337 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6c2307b8e64c2481d5bd47e9189fea (DEFAULT_TABLE table=default.Table_7e0f5faa00554a3b9cdaff8e346d048a [id=d34e9f2f945748988e15f9ae5a981b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.475684 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fde27a1501fd4d958d10183826b7ec3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.476490 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fde27a1501fd4d958d10183826b7ec3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.483291 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927d4359637f4ec189a5143e8201c7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.484486 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.484609 28604 ts_tablet_manager.cc:1398] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.088s user 0.002s sys 0.002s
I20240503 14:12:14.485462 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c2307b8e64c2481d5bd47e9189fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.488883 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f226db2fdb47ba9472a3aa65b953c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.492290 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f226db2fdb47ba9472a3aa65b953c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.492601 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f1f5216cc940888803ce8f6acd2a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.492930 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd037e135904e2cab09cbf39fed0b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.493680 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c0540960ac45aabeaf13f888caf238 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.495273 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 752b369f24224f2486834814a9d08495) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.495064 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.497454 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1ddf62892e2149ccb722a65a606c43e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.498065 28603 ts_tablet_manager.cc:1909] T 0bd5234791524eddb7edd493e39fb993 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:14.498222 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ddf62892e2149ccb722a65a606c43e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.500922 29431 raft_consensus.cc:482] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.502445 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb85d6e65b04937b3f89f8dcccd6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.502939 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:14.503556 28427 consensus_peers.cc:485] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7945118bb17b431dac8670007d2d0a18. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:14.506301 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9d878dfb144e5db914f8619c13ce8c (DEFAULT_TABLE table=default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.503571 29431 raft_consensus.cc:504] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.507447 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6c2307b8e64c2481d5bd47e9189fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.508569 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c0540960ac45aabeaf13f888caf238 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.510107 28604 tablet_bootstrap.cc:492] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:14.510113 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.511452 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b139ab0b5864024ac48215d46b0a7bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:14.517153 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9d878dfb144e5db914f8619c13ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.518795 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0bd5234791524eddb7edd493e39fb993 (table default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]) successfully deleted
W20240503 14:12:14.519150 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6b94092725fa4956a4c00e3685f9aeaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b94092725fa4956a4c00e3685f9aeaf already in progress: creating tablet
W20240503 14:12:14.519421 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet aa3ce05b422a4166bfac468acb84e4cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa3ce05b422a4166bfac468acb84e4cc already in progress: creating tablet
W20240503 14:12:14.519649 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 4fad4508a29d4822b81af6947b57e7ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fad4508a29d4822b81af6947b57e7ef already in progress: creating tablet
I20240503 14:12:14.521785 28603 tablet_replica.cc:330] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.521982 28603 raft_consensus.cc:2229] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.522127 28603 raft_consensus.cc:2258] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.522913 28603 ts_tablet_manager.cc:1875] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:14.517978 28427 leader_election.cc:341] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:14.524535 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd037e135904e2cab09cbf39fed0b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.526633 29431 leader_election.cc:288] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:14.537358 28603 ts_tablet_manager.cc:1888] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.537629 28603 log.cc:1196] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0b255d3e3dc64fb69c2ef6ed389db6a6
I20240503 14:12:14.538254 28603 ts_tablet_manager.cc:1909] T 0b255d3e3dc64fb69c2ef6ed389db6a6 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.538910 28603 tablet_replica.cc:330] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.539168 28603 raft_consensus.cc:2229] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.539436 28603 raft_consensus.cc:2258] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.541417 28603 ts_tablet_manager.cc:1875] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:14.545322 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c04707221b5241db8cd380ad5db24af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c04707221b5241db8cd380ad5db24af5 already in progress: creating tablet
I20240503 14:12:14.546243 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0b255d3e3dc64fb69c2ef6ed389db6a6 (table default.Table_068cea3b090241c7980d0edb57c05aec [id=80b934d1638444ebb855ebacf917c23e]) successfully deleted
I20240503 14:12:14.546692 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0993be031cc4792a9eadba9ce41d024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.548869 28604 tablet_bootstrap.cc:654] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.552460 29480 raft_consensus.cc:504] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.562635 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8894e326ff3a419d82f7e466b85bf687" } new_table_name: "default.Table_cf24dd3520bd4373acfd719f2c7737ad" modify_external_catalogs: true
I20240503 14:12:14.565204 28603 ts_tablet_manager.cc:1888] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:14.565330 28603 log.cc:1196] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/8c7d57fc4486457e9848daad79f04356
I20240503 14:12:14.566737 28603 ts_tablet_manager.cc:1909] T 8c7d57fc4486457e9848daad79f04356 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.566666 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c895a23bc5d64550b963a2b67358a358" } new_table_name: "default.Table_8ea2f91bb9e242aeb07b426cd711d537" modify_external_catalogs: true
I20240503 14:12:14.567524 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 8c7d57fc4486457e9848daad79f04356 (table default.Table_ea4ed2f332dc46aaa1f6e7025a1a1c3c [id=5cf73f88a73a4cc5966467ecfff2140f]) successfully deleted
W20240503 14:12:14.568399 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c9c0540960ac45aabeaf13f888caf238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c0540960ac45aabeaf13f888caf238 already in progress: creating tablet
I20240503 14:12:14.568392 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b8a25247ce2a4633b8e8bc77291fb69d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.569044 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8a25247ce2a4633b8e8bc77291fb69d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:14.569391 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 already in progress: creating tablet
W20240503 14:12:14.570271 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 920be6d87c4440d0bf7453c86cfe0b5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920be6d87c4440d0bf7453c86cfe0b5c already in progress: creating tablet
W20240503 14:12:14.571081 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 77f0081cc2cb48aaa368741109032fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f0081cc2cb48aaa368741109032fd1 already in progress: creating tablet
I20240503 14:12:14.576519 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet 1603e2360041427f8635782e812b6431 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:14.580838 28614 raft_consensus.cc:729] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.581161 28614 consensus_queue.cc:260] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.581650 28614 ts_tablet_manager.cc:1398] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.178s user 0.016s sys 0.000s
I20240503 14:12:14.581923 28614 tablet_bootstrap.cc:492] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:14.583107 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f41bb9c5d40b0913264864ecc3528" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:14.583308 28351 raft_consensus.cc:3037] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.584977 28614 tablet_bootstrap.cc:654] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.586375 28351 raft_consensus.cc:2454] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:14.593632 29281 raft_consensus.cc:482] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.594955 28614 tablet_bootstrap.cc:492] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:14.596413 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f3b31439f5e6426cb04a9110f6c93f27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.597082 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b31439f5e6426cb04a9110f6c93f27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.599560 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3f5415973b9641a3a612e9236aad1e92" } new_table_name: "default.Table_8bbb6d64decd4c3897319ca3e7130213" modify_external_catalogs: true
I20240503 14:12:14.601286 28614 ts_tablet_manager.cc:1367] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 14:12:14.602453 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f41bb9c5d40b0913264864ecc3528 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.602730 28603 tablet_replica.cc:330] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.602913 28603 raft_consensus.cc:2229] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.603055 28603 raft_consensus.cc:2258] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.603982 28603 ts_tablet_manager.cc:1875] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:14.604467 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f0b1e4dd193a4d6aa74cbc8280c025b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.605090 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0b1e4dd193a4d6aa74cbc8280c025b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.608726 28614 raft_consensus.cc:348] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.608335 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet ea261d8d5d174bee804aa2e171f35008 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:14.608418 28150 leader_election.cc:302] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:14.611266 28614 raft_consensus.cc:374] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.611357 28614 raft_consensus.cc:729] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.611449 28603 ts_tablet_manager.cc:1888] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.611537 28603 log.cc:1196] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7c6f41bb9c5d40b0913264864ecc3528
I20240503 14:12:14.611617 28614 consensus_queue.cc:260] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.611928 28603 ts_tablet_manager.cc:1909] T 7c6f41bb9c5d40b0913264864ecc3528 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.613633 28614 ts_tablet_manager.cc:1398] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 14:12:14.614267 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7c6f41bb9c5d40b0913264864ecc3528 (table default.Table_6add22bd3ca9452fa98a991f4e9b817f [id=806d4f0bc4524d9481487418ff797a5d]) successfully deleted
I20240503 14:12:14.614822 28614 tablet_bootstrap.cc:492] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:14.618031 28614 tablet_bootstrap.cc:654] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.618090 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 64f226db2fdb47ba9472a3aa65b953c3 (DEFAULT_TABLE table=default.Table_4e0a1dcdeddb46cfac04de76adfc27e5 [id=c53d59b2f4634fd7b3c26f749d9acc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.624455 28327 tablet_service.cc:1467] Processing CreateTablet for tablet bab214b7ec58425d8cd1180ee16b273d (DEFAULT_TABLE table=default.Table_b80186a1b50741aca2ee3ff55a4ef6be [id=fb38b20d1471483a974d60a54ae2afa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.624913 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab214b7ec58425d8cd1180ee16b273d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.626431 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd5234791524eddb7edd493e39fb993 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b21cd1262d94d1c994b29c44f86cb5e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.628412 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 11d9f47543524fc5921c99aad869035f (DEFAULT_TABLE table=default.Table_98e9ee5715624c0c804e368311ceabb5 [id=2c12e9e728b64cb9a5e460b134b0f528]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.628845 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d9f47543524fc5921c99aad869035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.630008 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4929b450373c4b689562f4a1d9e4b4f2" } modify_external_catalogs: true
I20240503 14:12:14.630100 28328 tablet_service.cc:1467] Processing CreateTablet for tablet ff07bd9bd59f4b13955904cc172c044c (DEFAULT_TABLE table=default.Table_b1ae9903413045af9a9fa56d22ac1552 [id=b040f9096f95497a8c418a5e6795c21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.630234 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4929b450373c4b689562f4a1d9e4b4f2" } modify_external_catalogs: true
I20240503 14:12:14.630462 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff07bd9bd59f4b13955904cc172c044c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.632418 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc05dfe2b324ca787ca33f52be6a308 (DEFAULT_TABLE table=default.Table_9f158ace156f44beacdd67afbd7eafc0 [id=a62852b5903843ef8bba8056ce57d485]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.632773 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc05dfe2b324ca787ca33f52be6a308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.633697 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf0756e71ab448a979e7bb85399b42c (DEFAULT_TABLE table=default.Table_daa40e7d4485405eabf710a2ddc0a60e [id=bf78cfab3d1e494fac3ddef63830bbcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.635553 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 962df44408484e3f8d7b7138f27c1a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e4a544fc27624dcc8a4cb003e59408f2) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.635962 28329 tablet_service.cc:1467] Processing CreateTablet for tablet bbc90d9b37bc4c57af3d29ce0f7da287 (DEFAULT_TABLE table=default.Table_d5c70a4285c24a50b556481cf3793c19 [id=3ff02b51d6f145c1b64acfa0e0aae313]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.636323 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc90d9b37bc4c57af3d29ce0f7da287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.636899 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 833ad747feca411aa480dd420a6c02cc (DEFAULT_TABLE table=default.Table_7aba6cf95a4e4459aa96ea157115e3d2 [id=5d5b8aca478d48a287e7b9795de1551c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.637180 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf0756e71ab448a979e7bb85399b42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.637403 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 4d912da2c5c546dfa7d2c64b5fc535c8 (DEFAULT_TABLE table=default.Table_6ec7fc5bf73b4e3aa4da80a76c3b2611 [id=cbe47134ece44778ada089c0d24e31eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.637694 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d912da2c5c546dfa7d2c64b5fc535c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.640745 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b509f93207e3417c9bfe50c97d92096e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.641465 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b509f93207e3417c9bfe50c97d92096e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.640339 29281 raft_consensus.cc:504] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.640019 28464 tablet_service.cc:1467] Processing CreateTablet for tablet e21a5dafa82f4660927d96c779a23478 (DEFAULT_TABLE table=default.Table_cc8ca3c970a949fab24298af4538c323 [id=ee1e18b4b41042199f351112b57335b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.644209 28603 tablet_replica.cc:330] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.644418 28603 raft_consensus.cc:2229] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.644618 28603 raft_consensus.cc:2258] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.647439 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833ad747feca411aa480dd420a6c02cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.651247 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff1958dc488e4854a8c8e74da9fab243 consensus state has no leader
I20240503 14:12:14.651937 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_dcf7f653c09f4112b516cee572ea9993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.652734 28603 ts_tablet_manager.cc:1875] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:14.652868 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcf7f653c09f4112b516cee572ea9993 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:14.654574 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31ddd791d4e14630bfb3b4b1513f76e1 consensus state has no leader
I20240503 14:12:14.655702 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2221218847554cbd9d3242a9a6431e46
I20240503 14:12:14.656842 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221218847554cbd9d3242a9a6431e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:14.657166 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8acbc154c03c47c1b2f9f433fcb2ef9a consensus state has no leader
I20240503 14:12:14.662958 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ea261d8d5d174bee804aa2e171f35008
I20240503 14:12:14.663769 28031 catalog_manager.cc:6554] ReplaceTablet: tablet ea261d8d5d174bee804aa2e171f35008 deleted and replaced by tablet ae05e174ccbd4a588cf7ffb1de42632e
I20240503 14:12:14.650691 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21a5dafa82f4660927d96c779a23478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.664515 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet ea261d8d5d174bee804aa2e171f35008 with delete_type TABLET_DATA_DELETED (replaced by tablet ae05e174ccbd4a588cf7ffb1de42632e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.663282 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 833ad747feca411aa480dd420a6c02cc (DEFAULT_TABLE table=default.Table_7aba6cf95a4e4459aa96ea157115e3d2 [id=5d5b8aca478d48a287e7b9795de1551c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.664798 28469 tablet_service.cc:1467] Processing CreateTablet for tablet bbc90d9b37bc4c57af3d29ce0f7da287 (DEFAULT_TABLE table=default.Table_d5c70a4285c24a50b556481cf3793c19 [id=3ff02b51d6f145c1b64acfa0e0aae313]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.663213 28463 tablet_service.cc:1467] Processing CreateTablet for tablet ff07bd9bd59f4b13955904cc172c044c (DEFAULT_TABLE table=default.Table_b1ae9903413045af9a9fa56d22ac1552 [id=b040f9096f95497a8c418a5e6795c21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.665277 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833ad747feca411aa480dd420a6c02cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.665238 28473 tablet_service.cc:1467] Processing CreateTablet for tablet bab214b7ec58425d8cd1180ee16b273d (DEFAULT_TABLE table=default.Table_b80186a1b50741aca2ee3ff55a4ef6be [id=fb38b20d1471483a974d60a54ae2afa3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.665863 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62b7129591554425bc6c23af414fbffb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.667332 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff07bd9bd59f4b13955904cc172c044c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.668726 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221218847554cbd9d3242a9a6431e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.673552 28603 ts_tablet_manager.cc:1888] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.670665 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6dfb1a277c824cbfad52ce51179289a6" } new_table_name: "default.Table_3bf21cdd53304e70a9c5b910bc33ce19" modify_external_catalogs: true
I20240503 14:12:14.674315 28603 log.cc:1196] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/962df44408484e3f8d7b7138f27c1a66
I20240503 14:12:14.674779 28603 ts_tablet_manager.cc:1909] T 962df44408484e3f8d7b7138f27c1a66 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.671370 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c297c309fedc4148849b57c91f68d1b4" } new_table_name: "default.Table_501a1223ca764fdb8381a08a543f6bd7" modify_external_catalogs: true
I20240503 14:12:14.671921 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6d7202481c4747ddb44a78ba8bb0063f" } modify_external_catalogs: true
I20240503 14:12:14.677502 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6d7202481c4747ddb44a78ba8bb0063f" } modify_external_catalogs: true
I20240503 14:12:14.672350 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8ea2f91bb9e242aeb07b426cd711d537" } modify_external_catalogs: true
I20240503 14:12:14.678768 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8ea2f91bb9e242aeb07b426cd711d537" } modify_external_catalogs: true
I20240503 14:12:14.672919 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c7035b3f783948498dfb7bd5ed14aad8" } modify_external_catalogs: true
I20240503 14:12:14.679450 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c7035b3f783948498dfb7bd5ed14aad8" } modify_external_catalogs: true
I20240503 14:12:14.673275 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_21ea716e646a43279621abff726dedd1" } modify_external_catalogs: true
I20240503 14:12:14.683256 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_21ea716e646a43279621abff726dedd1" } modify_external_catalogs: true
I20240503 14:12:14.683923 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 962df44408484e3f8d7b7138f27c1a66 (table default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]) successfully deleted
I20240503 14:12:14.685691 29281 leader_election.cc:288] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:14.686810 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2221218847554cbd9d3242a9a6431e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2221218847554cbd9d3242a9a6431e46 already in progress: creating tablet
W20240503 14:12:14.687687 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ea261d8d5d174bee804aa2e171f35008 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea261d8d5d174bee804aa2e171f35008 already in progress: creating tablet
I20240503 14:12:14.687743 28192 tablet_service.cc:1467] Processing CreateTablet for tablet ed177731c989482ebda1694dabcb8fb8 (DEFAULT_TABLE table=default.Table_3f6ca1c2e7314e7ebb2235561072ed1c [id=b396f957941242c7a09ef0bd312ff25c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.691391 28653 log.cc:1196] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/1d77014c66624f958b38f7cc83644891
I20240503 14:12:14.691543 28614 tablet_bootstrap.cc:492] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:14.691776 28614 ts_tablet_manager.cc:1367] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.009s
I20240503 14:12:14.692739 28614 raft_consensus.cc:348] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.693068 28614 raft_consensus.cc:374] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.693171 28614 raft_consensus.cc:729] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.693457 28614 consensus_queue.cc:260] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.694159 28614 ts_tablet_manager.cc:1398] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:14.694478 28614 tablet_bootstrap.cc:492] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:14.697690 28614 tablet_bootstrap.cc:654] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.704172 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 239f14be29e54cd29bb5e96622aacd55 (DEFAULT_TABLE table=default.Table_15571f0ea45241568811001dfead7fff [id=f62601475c2b474a8ee939f93e546dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.706286 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239f14be29e54cd29bb5e96622aacd55. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.707093 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac577af6460b423288a790e532b00f82 consensus state has no leader
I20240503 14:12:14.709352 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.709856 28603 tablet_replica.cc:330] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.710072 28603 raft_consensus.cc:2229] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.710235 28603 raft_consensus.cc:2258] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.711207 28603 ts_tablet_manager.cc:1875] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:14.712363 28183 tablet_service.cc:1467] Processing CreateTablet for tablet b0993be031cc4792a9eadba9ce41d024 (DEFAULT_TABLE table=default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.714426 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a6a58db50f849a8be729c10dbe03e76 consensus state has no leader
I20240503 14:12:14.717981 28600 tablet_bootstrap.cc:654] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.720171 29281 raft_consensus.cc:2781] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.721171 29480 leader_election.cc:288] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:14.722173 28603 ts_tablet_manager.cc:1888] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.722386 28603 log.cc:1196] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ca6627f0d9a544bb8176bea6c1af5ab4
I20240503 14:12:14.722766 28603 ts_tablet_manager.cc:1909] T ca6627f0d9a544bb8176bea6c1af5ab4 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.724651 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ca6627f0d9a544bb8176bea6c1af5ab4 (table default.Table_e3e1acbcbb7c4aeab3db92cc253b441d [id=9ac543611bee46c08fb673c287aa6af8]) successfully deleted
I20240503 14:12:14.730800 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed177731c989482ebda1694dabcb8fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.735476 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 812d20a15f3b4fa1ad43851b7aea56bc (DEFAULT_TABLE table=default.Table_6b41ac2dcacf495bbb627aa567c42764 [id=67119fd7390e4bf09a4b207827ac87d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.735975 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812d20a15f3b4fa1ad43851b7aea56bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.737917 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 132134us
I20240503 14:12:14.738806 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:14.741358 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb85d6e65b04937b3f89f8dcccd6df2 (DEFAULT_TABLE table=default.Table_d4a44441aa4647449f447922587e6ffd [id=3311b8b2dcf34b8583210f2e08eaca36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.741750 28330 tablet_service.cc:1467] Processing CreateTablet for tablet ad9baf131a1948d8ae5fae4c706aa726 (DEFAULT_TABLE table=default.Table_fde27a1501fd4d958d10183826b7ec3d [id=1ad032e486be483d89c646f6a779d2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.742199 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9baf131a1948d8ae5fae4c706aa726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.742729 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd037e135904e2cab09cbf39fed0b13 (DEFAULT_TABLE table=default.Table_d121667544004d9e83e4daf3f58f7473 [id=7877e3c7f4b14fcebfd7bec2e15b6240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.745518 28653 ts_tablet_manager.cc:1909] T 1d77014c66624f958b38f7cc83644891 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:14.745645 28017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.50.253:44259: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:12:14.746699 28150 leader_election.cc:341] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:14.747782 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 8d65fdf69c6846f98f2e80e6101fffc4 (DEFAULT_TABLE table=default.Table_f3b31439f5e6426cb04a9110f6c93f27 [id=8812129023e84d20b03408f44816f04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.748387 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d65fdf69c6846f98f2e80e6101fffc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.752996 28327 tablet_service.cc:1467] Processing CreateTablet for tablet a35224f4ad6a4eb78202f8dc44056d38 (DEFAULT_TABLE table=default.Table_82dca5d2c206486786a35bb34369e2d7 [id=d65b3ebfa6b64919b918ffdee938b42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.753649 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 2507b459d2884e27b20a77ac3017e770 (DEFAULT_TABLE table=default.Table_1efd634aa4c0450283b143ce587da86e [id=e75ec34373fd4e3da4d140ae98ce6db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.754287 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35224f4ad6a4eb78202f8dc44056d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.754654 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2507b459d2884e27b20a77ac3017e770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.758591 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 752b369f24224f2486834814a9d08495 (DEFAULT_TABLE table=default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.759073 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752b369f24224f2486834814a9d08495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.772423 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7d57fc4486457e9848daad79f04356 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.764861 28193 tablet_service.cc:1467] Processing CreateTablet for tablet e21a5dafa82f4660927d96c779a23478 (DEFAULT_TABLE table=default.Table_cc8ca3c970a949fab24298af4538c323 [id=ee1e18b4b41042199f351112b57335b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.777794 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:14.780790 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet b59945d011b64cba87c854e2ffde8226 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:14.785795 28614 tablet_bootstrap.cc:492] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:14.785964 28614 ts_tablet_manager.cc:1367] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.013s
I20240503 14:12:14.786793 28614 raft_consensus.cc:348] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.787014 28614 raft_consensus.cc:374] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.787107 28614 raft_consensus.cc:729] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.787331 28614 consensus_queue.cc:260] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.787601 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 752b369f24224f2486834814a9d08495) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.792783 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc05dfe2b324ca787ca33f52be6a308 (DEFAULT_TABLE table=default.Table_9f158ace156f44beacdd67afbd7eafc0 [id=a62852b5903843ef8bba8056ce57d485]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.793365 28661 ts_tablet_manager.cc:1888] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.793280 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.793540 28661 log.cc:1196] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/bd76060daab84c2f9f6727b8a9bb3a49
I20240503 14:12:14.794119 28661 ts_tablet_manager.cc:1909] T bd76060daab84c2f9f6727b8a9bb3a49 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:14.794415 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6371e68ee85a47cdb0e8e21f7124d94a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6371e68ee85a47cdb0e8e21f7124d94a already in progress: creating tablet
I20240503 14:12:14.795121 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 498) took 1555 ms. Trace:
I20240503 14:12:14.795272 28661 rpcz_store.cc:276] 0503 14:12:13.239245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.239368 (+ 123us) service_pool.cc:225] Handling call
0503 14:12:14.392737 (+1153369us) ts_tablet_manager.cc:1120] Deleting tablet bd76060daab84c2f9f6727b8a9bb3a49
0503 14:12:14.392750 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.529915 (+137165us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:14.667073 (+137158us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:14.795067 (+127994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153104}
I20240503 14:12:14.795742 28661 tablet_replica.cc:330] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:14.796180 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 11d9f47543524fc5921c99aad869035f (DEFAULT_TABLE table=default.Table_98e9ee5715624c0c804e368311ceabb5 [id=2c12e9e728b64cb9a5e460b134b0f528]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.796664 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab214b7ec58425d8cd1180ee16b273d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.796981 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc90d9b37bc4c57af3d29ce0f7da287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.797029 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9d878dfb144e5db914f8619c13ce8c (DEFAULT_TABLE table=default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.797520 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0d81fa49986c4ca99528a823b4a22ae6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.798749 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d81fa49986c4ca99528a823b4a22ae6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.798673 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:14.799253 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet ea261d8d5d174bee804aa2e171f35008 with delete_type TABLET_DATA_DELETED (replaced by tablet ae05e174ccbd4a588cf7ffb1de42632e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.800310 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet f0287bc315644f0aaa976a56174576d9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:14.802670 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_84ebaac5471744d0ad8b983a03565cee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.803016 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 4d912da2c5c546dfa7d2c64b5fc535c8 (DEFAULT_TABLE table=default.Table_6ec7fc5bf73b4e3aa4da80a76c3b2611 [id=cbe47134ece44778ada089c0d24e31eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.803140 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f226db2fdb47ba9472a3aa65b953c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.792204 29431 raft_consensus.cc:482] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:14.803428 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84ebaac5471744d0ad8b983a03565cee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.804054 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 239f14be29e54cd29bb5e96622aacd55 (DEFAULT_TABLE table=default.Table_15571f0ea45241568811001dfead7fff [id=f62601475c2b474a8ee939f93e546dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.796039 28661 raft_consensus.cc:2229] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:14.806800 29431 raft_consensus.cc:504] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.807600 29431 leader_election.cc:288] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:14.797132 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf0756e71ab448a979e7bb85399b42c (DEFAULT_TABLE table=default.Table_daa40e7d4485405eabf710a2ddc0a60e [id=bf78cfab3d1e494fac3ddef63830bbcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.808422 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d9f47543524fc5921c99aad869035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.809626 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9d878dfb144e5db914f8619c13ce8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.814450 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c0540960ac45aabeaf13f888caf238 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.816797 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 812d20a15f3b4fa1ad43851b7aea56bc (DEFAULT_TABLE table=default.Table_6b41ac2dcacf495bbb627aa567c42764 [id=67119fd7390e4bf09a4b207827ac87d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.817018 28614 ts_tablet_manager.cc:1398] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20240503 14:12:14.814927 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:14.818921 28150 leader_election.cc:341] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:14.819119 28150 leader_election.cc:302] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:14.819247 28614 tablet_bootstrap.cc:492] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:14.819226 28661 raft_consensus.cc:2258] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.790333 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.821422 28614 tablet_bootstrap.cc:654] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.821767 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f41bb9c5d40b0913264864ecc3528" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:14.822042 28491 raft_consensus.cc:3037] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.822018 29503 raft_consensus.cc:2726] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:14.823094 28474 tablet_service.cc:1467] Processing CreateTablet for tablet a35224f4ad6a4eb78202f8dc44056d38 (DEFAULT_TABLE table=default.Table_82dca5d2c206486786a35bb34369e2d7 [id=d65b3ebfa6b64919b918ffdee938b42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.823864 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d912da2c5c546dfa7d2c64b5fc535c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.825230 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3cbcae22ea124edaa14c9409f3052714
I20240503 14:12:14.824183 28471 tablet_service.cc:1467] Processing CreateTablet for tablet ad9baf131a1948d8ae5fae4c706aa726 (DEFAULT_TABLE table=default.Table_fde27a1501fd4d958d10183826b7ec3d [id=1ad032e486be483d89c646f6a779d2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.825362 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 8d65fdf69c6846f98f2e80e6101fffc4 (DEFAULT_TABLE table=default.Table_f3b31439f5e6426cb04a9110f6c93f27 [id=8812129023e84d20b03408f44816f04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.825908 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35224f4ad6a4eb78202f8dc44056d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.827746 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf0756e71ab448a979e7bb85399b42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.826349 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbcae22ea124edaa14c9409f3052714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.827893 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.828410 28603 tablet_replica.cc:330] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.828601 28603 raft_consensus.cc:2229] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.828675 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc05dfe2b324ca787ca33f52be6a308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.828784 28603 raft_consensus.cc:2258] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:14.829310 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 3cbcae22ea124edaa14c9409f3052714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cbcae22ea124edaa14c9409f3052714 already in progress: creating tablet
I20240503 14:12:14.830408 28603 ts_tablet_manager.cc:1875] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:14.830753 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 95c97a43bd714da68c1a77408f3d1cd1
I20240503 14:12:14.830806 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d65fdf69c6846f98f2e80e6101fffc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.831597 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239f14be29e54cd29bb5e96622aacd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.831583 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.832255 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812d20a15f3b4fa1ad43851b7aea56bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.831869 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ec78ac6bb77c4e4790e3d2c0f107c6e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:14.833680 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9baf131a1948d8ae5fae4c706aa726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.834254 28182 tablet_service.cc:1467] Processing CreateTablet for tablet 11d9f47543524fc5921c99aad869035f (DEFAULT_TABLE table=default.Table_98e9ee5715624c0c804e368311ceabb5 [id=2c12e9e728b64cb9a5e460b134b0f528]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:14.837648 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec78ac6bb77c4e4790e3d2c0f107c6e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.840631 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 2507b459d2884e27b20a77ac3017e770 (DEFAULT_TABLE table=default.Table_1efd634aa4c0450283b143ce587da86e [id=e75ec34373fd4e3da4d140ae98ce6db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.842243 28614 tablet_bootstrap.cc:492] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:14.842478 28614 ts_tablet_manager.cc:1367] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240503 14:12:14.842533 29503 raft_consensus.cc:482] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:14.843353 28614 raft_consensus.cc:348] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.843564 28614 raft_consensus.cc:374] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.843631 28614 raft_consensus.cc:729] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.843932 28614 consensus_queue.cc:260] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.847496 28614 ts_tablet_manager.cc:1398] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:14.848104 28614 tablet_bootstrap.cc:492] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:14.848273 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd037e135904e2cab09cbf39fed0b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.848605 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21a5dafa82f4660927d96c779a23478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.851379 28614 tablet_bootstrap.cc:654] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.851329 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 752b369f24224f2486834814a9d08495 (DEFAULT_TABLE table=default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.852252 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb85d6e65b04937b3f89f8dcccd6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.855710 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d9f47543524fc5921c99aad869035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.856256 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0993be031cc4792a9eadba9ce41d024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.856875 28661 ts_tablet_manager.cc:1875] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:14.857057 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2507b459d2884e27b20a77ac3017e770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.859753 29281 raft_consensus.cc:686] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:14.861583 28603 ts_tablet_manager.cc:1888] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.861708 28603 log.cc:1196] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6371e68ee85a47cdb0e8e21f7124d94a
I20240503 14:12:14.862018 28603 ts_tablet_manager.cc:1909] T 6371e68ee85a47cdb0e8e21f7124d94a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.847616 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc05dfe2b324ca787ca33f52be6a308 (DEFAULT_TABLE table=default.Table_9f158ace156f44beacdd67afbd7eafc0 [id=a62852b5903843ef8bba8056ce57d485]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.864918 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752b369f24224f2486834814a9d08495. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.865763 28427 consensus_peers.cc:485] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:14.839599 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ff1958dc488e4854a8c8e74da9fab243
I20240503 14:12:14.848106 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet bd76060daab84c2f9f6727b8a9bb3a49 (table default.Table_3668987e06c74fa6bc566d6e46f92879 [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
I20240503 14:12:14.866741 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6371e68ee85a47cdb0e8e21f7124d94a (table default.Table_355c3b14ae4f480dba47c2c90dc9dba2 [id=d7eac407621f4b03b21439fc2b0f7dc3]) successfully deleted
W20240503 14:12:14.867098 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 95c97a43bd714da68c1a77408f3d1cd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c97a43bd714da68c1a77408f3d1cd1 already in progress: creating tablet
I20240503 14:12:14.867835 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1958dc488e4854a8c8e74da9fab243 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:14.868598 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ff1958dc488e4854a8c8e74da9fab243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1958dc488e4854a8c8e74da9fab243 already in progress: creating tablet
I20240503 14:12:14.869167 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044f34588d045ce898435b61a90503f with delete_type TABLET_DATA_DELETED (replaced by tablet daf5d153ab5142c0902cc6e96629f412) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.869491 28603 tablet_replica.cc:330] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:14.869735 28603 raft_consensus.cc:2229] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.869983 28603 raft_consensus.cc:2258] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:14.873061 28427 leader_election.cc:341] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 63e977f7837e49fb93ef0fd18b619792
I20240503 14:12:14.874459 28186 tablet_service.cc:1467] Processing CreateTablet for tablet bab214b7ec58425d8cd1180ee16b273d (DEFAULT_TABLE table=default.Table_b80186a1b50741aca2ee3ff55a4ef6be [id=fb38b20d1471483a974d60a54ae2afa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.875744 28603 ts_tablet_manager.cc:1875] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:14.877985 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b59945d011b64cba87c854e2ffde8226
I20240503 14:12:14.878626 28027 catalog_manager.cc:6554] ReplaceTablet: tablet b59945d011b64cba87c854e2ffde8226 deleted and replaced by tablet 47965a978a904f2d876690de01f89771
I20240503 14:12:14.879248 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet b59945d011b64cba87c854e2ffde8226 with delete_type TABLET_DATA_DELETED (replaced by tablet 47965a978a904f2d876690de01f89771) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.881366 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c89fe31dfbac4d48a061a21a358b84f6" } new_table_name: "default.Table_d9142b8d36e542648c29cf7dcce32d4c" modify_external_catalogs: true
I20240503 14:12:14.881286 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9d3155335c5a46209c736c3c3ccd5eb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.881963 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d3155335c5a46209c736c3c3ccd5eb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.882230 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbcae22ea124edaa14c9409f3052714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.882596 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:14.883152 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7e0f5faa00554a3b9cdaff8e346d048a" } modify_external_catalogs: true
I20240503 14:12:14.883349 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7e0f5faa00554a3b9cdaff8e346d048a" } modify_external_catalogs: true
I20240503 14:12:14.884238 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9f6a864cf34b498f87ccf5afca7d29e8" } new_table_name: "default.Table_f3380746b00d4e5f9ba9f5851075ec37" modify_external_catalogs: true
I20240503 14:12:14.886525 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f0287bc315644f0aaa976a56174576d9
I20240503 14:12:14.887312 28024 catalog_manager.cc:6554] ReplaceTablet: tablet f0287bc315644f0aaa976a56174576d9 deleted and replaced by tablet 6a27d21de6a6441d810f379bba88f9a2
I20240503 14:12:14.887477 28491 raft_consensus.cc:2454] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:14.887828 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet f0287bc315644f0aaa976a56174576d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a27d21de6a6441d810f379bba88f9a2) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:14.888640 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ba1e59290eb64bd488b68e68cb9b2548" } new_table_name: "default.Table_7fbc01371cd145eeb7b51110d14f6456" modify_external_catalogs: true
I20240503 14:12:14.890655 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4e13e0a790bc495f93e8cabc45270a80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:14.891353 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e13e0a790bc495f93e8cabc45270a80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:14.901680 28603 ts_tablet_manager.cc:1888] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:14.903539 28603 log.cc:1196] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/1044f34588d045ce898435b61a90503f
I20240503 14:12:14.904449 28603 ts_tablet_manager.cc:1909] T 1044f34588d045ce898435b61a90503f P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:14.905294 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 1044f34588d045ce898435b61a90503f (table default.Table_6673b8d0a18d42c3a18f6e611c596580 [id=3c43a0fcf51143179a1ed0e1b8e333c1]) successfully deleted
W20240503 14:12:14.906450 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b59945d011b64cba87c854e2ffde8226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b59945d011b64cba87c854e2ffde8226 already in progress: creating tablet
W20240503 14:12:14.907703 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f0287bc315644f0aaa976a56174576d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0287bc315644f0aaa976a56174576d9 already in progress: creating tablet
I20240503 14:12:14.906574 29281 consensus_queue.cc:237] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.909574 29503 raft_consensus.cc:504] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:14.910609 29503 leader_election.cc:288] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:14.910437 28427 consensus_peers.cc:485] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:14.912220 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf0756e71ab448a979e7bb85399b42c (DEFAULT_TABLE table=default.Table_daa40e7d4485405eabf710a2ddc0a60e [id=bf78cfab3d1e494fac3ddef63830bbcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.922739 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221218847554cbd9d3242a9a6431e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.926051 28179 tablet_service.cc:1467] Processing CreateTablet for tablet bbc90d9b37bc4c57af3d29ce0f7da287 (DEFAULT_TABLE table=default.Table_d5c70a4285c24a50b556481cf3793c19 [id=3ff02b51d6f145c1b64acfa0e0aae313]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.930238 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 8d65fdf69c6846f98f2e80e6101fffc4 (DEFAULT_TABLE table=default.Table_f3b31439f5e6426cb04a9110f6c93f27 [id=8812129023e84d20b03408f44816f04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.942579 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 812d20a15f3b4fa1ad43851b7aea56bc (DEFAULT_TABLE table=default.Table_6b41ac2dcacf495bbb627aa567c42764 [id=67119fd7390e4bf09a4b207827ac87d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.945832 28182 tablet_service.cc:1467] Processing CreateTablet for tablet 752b369f24224f2486834814a9d08495 (DEFAULT_TABLE table=default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.948750 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 429) took 3176 ms. Trace:
I20240503 14:12:14.950093 28653 rpcz_store.cc:276] 0503 14:12:11.620793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.624143 (+ 3350us) service_pool.cc:225] Handling call
0503 14:12:13.930741 (+2306598us) ts_tablet_manager.cc:1120] Deleting tablet 1d77014c66624f958b38f7cc83644891
0503 14:12:13.930754 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.292477 (+361723us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:14.387579 (+ 95102us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:14.796856 (+409277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2306300}
I20240503 14:12:14.951735 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 1d77014c66624f958b38f7cc83644891 (table default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]) successfully deleted
I20240503 14:12:14.951792 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 833ad747feca411aa480dd420a6c02cc (DEFAULT_TABLE table=default.Table_7aba6cf95a4e4459aa96ea157115e3d2 [id=5d5b8aca478d48a287e7b9795de1551c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.952327 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044f34588d045ce898435b61a90503f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:14.952592 28212 raft_consensus.cc:3037] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.955678 28212 raft_consensus.cc:2454] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:14.956494 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b139ab0b5864024ac48215d46b0a7bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:14.956583 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 4d912da2c5c546dfa7d2c64b5fc535c8 (DEFAULT_TABLE table=default.Table_6ec7fc5bf73b4e3aa4da80a76c3b2611 [id=cbe47134ece44778ada089c0d24e31eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.956797 28217 raft_consensus.cc:2454] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:14.957160 28150 consensus_peers.cc:485] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3514b79467dc42bc89fdacf8e39a000d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:14.957231 28196 tablet_service.cc:1467] Processing CreateTablet for tablet ff07bd9bd59f4b13955904cc172c044c (DEFAULT_TABLE table=default.Table_b1ae9903413045af9a9fa56d22ac1552 [id=b040f9096f95497a8c418a5e6795c21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.957533 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet ea261d8d5d174bee804aa2e171f35008 with delete_type TABLET_DATA_DELETED (replaced by tablet ae05e174ccbd4a588cf7ffb1de42632e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.957821 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.958091 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1958dc488e4854a8c8e74da9fab243 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.958209 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc05dfe2b324ca787ca33f52be6a308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.958281 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet b59945d011b64cba87c854e2ffde8226 with delete_type TABLET_DATA_DELETED (replaced by tablet 47965a978a904f2d876690de01f89771) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.958508 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet f0287bc315644f0aaa976a56174576d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a27d21de6a6441d810f379bba88f9a2) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.958616 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833ad747feca411aa480dd420a6c02cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.958868 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab214b7ec58425d8cd1180ee16b273d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.957345 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 55445.7us
I20240503 14:12:14.965209 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240abf8c2b2343408fc98a8dac1fdf09" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:14.966210 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240abf8c2b2343408fc98a8dac1fdf09" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:14.967164 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d912da2c5c546dfa7d2c64b5fc535c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.968281 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf0756e71ab448a979e7bb85399b42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.972499 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d65fdf69c6846f98f2e80e6101fffc4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:14.972745 28150 leader_election.cc:341] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:14.973008 28150 leader_election.cc:341] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:14.973202 28150 leader_election.cc:302] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:14.973677 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812d20a15f3b4fa1ad43851b7aea56bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.974169 29480 raft_consensus.cc:2726] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:14.978487 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff07bd9bd59f4b13955904cc172c044c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.984154 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752b369f24224f2486834814a9d08495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.984797 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 239f14be29e54cd29bb5e96622aacd55 (DEFAULT_TABLE table=default.Table_15571f0ea45241568811001dfead7fff [id=f62601475c2b474a8ee939f93e546dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.985344 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239f14be29e54cd29bb5e96622aacd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.989485 28193 tablet_service.cc:1467] Processing CreateTablet for tablet a35224f4ad6a4eb78202f8dc44056d38 (DEFAULT_TABLE table=default.Table_82dca5d2c206486786a35bb34369e2d7 [id=d65b3ebfa6b64919b918ffdee938b42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.989915 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35224f4ad6a4eb78202f8dc44056d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.990537 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 2507b459d2884e27b20a77ac3017e770 (DEFAULT_TABLE table=default.Table_1efd634aa4c0450283b143ce587da86e [id=e75ec34373fd4e3da4d140ae98ce6db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.990893 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2507b459d2884e27b20a77ac3017e770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.990984 28197 tablet_service.cc:1467] Processing CreateTablet for tablet ad9baf131a1948d8ae5fae4c706aa726 (DEFAULT_TABLE table=default.Table_fde27a1501fd4d958d10183826b7ec3d [id=1ad032e486be483d89c646f6a779d2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.991416 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbcae22ea124edaa14c9409f3052714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:14.991972 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9baf131a1948d8ae5fae4c706aa726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.016793 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 434) took 3018 ms. Trace:
I20240503 14:12:15.017066 28653 rpcz_store.cc:276] 0503 14:12:11.933265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:11.934229 (+ 964us) service_pool.cc:225] Handling call
0503 14:12:14.951281 (+3017052us) ts_tablet_manager.cc:1120] Deleting tablet 11ac912a3bd64cdc9082596e09c72e13
0503 14:12:14.951291 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.951533 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3016727}
W20240503 14:12:15.017927 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 11ac912a3bd64cdc9082596e09c72e13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11ac912a3bd64cdc9082596e09c72e13 already in progress: creating tablet
I20240503 14:12:15.018447 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc90d9b37bc4c57af3d29ce0f7da287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.019363 28600 tablet_bootstrap.cc:492] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.020925 28600 ts_tablet_manager.cc:1367] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.629s user 0.004s sys 0.007s
I20240503 14:12:15.020401 28653 tablet_replica.cc:330] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.022018 28653 raft_consensus.cc:2229] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.024428 28653 raft_consensus.cc:2258] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.027103 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:15.030987 28653 ts_tablet_manager.cc:1875] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.036636 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044f34588d045ce898435b61a90503f with delete_type TABLET_DATA_DELETED (replaced by tablet daf5d153ab5142c0902cc6e96629f412) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:15.037511 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1044f34588d045ce898435b61a90503f because the tablet was not found. No further retry: Not found: Tablet not found: 1044f34588d045ce898435b61a90503f
I20240503 14:12:15.037559 28600 raft_consensus.cc:348] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.041170 28600 raft_consensus.cc:374] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.041636 29503 raft_consensus.cc:482] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.041860 29503 raft_consensus.cc:504] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:15.042246 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 129146us
I20240503 14:12:15.041357 28600 raft_consensus.cc:729] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.043051 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:15.043746 28427 leader_election.cc:341] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 63e977f7837e49fb93ef0fd18b619792
I20240503 14:12:15.043915 28427 leader_election.cc:302] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.044297 29515 raft_consensus.cc:2726] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.044838 28427 leader_election.cc:302] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.044708 29317 consensus_queue.cc:1035] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.045192 29516 raft_consensus.cc:2781] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.045377 28427 leader_election.cc:302] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.045848 29317 raft_consensus.cc:2781] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.046000 29317 raft_consensus.cc:482] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.046135 29317 raft_consensus.cc:3037] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.046710 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6de7dc45944845628d02aa20bfa0dcad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.047372 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6de7dc45944845628d02aa20bfa0dcad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.047446 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1958dc488e4854a8c8e74da9fab243 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.048418 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
W20240503 14:12:15.048877 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09f1f5216cc940888803ce8f6acd2a47 consensus state has no leader
I20240503 14:12:15.049377 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b65e95f8090a40e5abb366c2aa904d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:15.049957 29317 raft_consensus.cc:504] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:15.050118 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b65e95f8090a40e5abb366c2aa904d60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.050699 29317 leader_election.cc:288] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:15.051244 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16fed718124a4640967ab13a1ac2b379 consensus state has no leader
I20240503 14:12:15.051560 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 2cad43f3c6094253ba87d0ed3fc29fd9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:15.052738 28604 tablet_bootstrap.cc:492] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.052918 28604 ts_tablet_manager.cc:1367] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.550s user 0.006s sys 0.006s
I20240503 14:12:15.054111 28604 raft_consensus.cc:348] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.054392 28604 raft_consensus.cc:374] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.054554 28604 raft_consensus.cc:729] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.054812 28604 consensus_queue.cc:260] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.056160 28604 ts_tablet_manager.cc:1398] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:15.049412 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a635bdd722254d13bf665ab69c76c554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.057003 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a635bdd722254d13bf665ab69c76c554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:15.059207 28150 leader_election.cc:341] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.060472 29503 leader_election.cc:288] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:15.064977 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 7927f3e0fb214441aaf52352b6956ef6 (DEFAULT_TABLE table=default.Table_dcf7f653c09f4112b516cee572ea9993 [id=eb7b7f9826ee4073969a65f9622567e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.055577 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet b59945d011b64cba87c854e2ffde8226 with delete_type TABLET_DATA_DELETED (replaced by tablet 47965a978a904f2d876690de01f89771) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.065395 28328 tablet_service.cc:1467] Processing CreateTablet for tablet ae05e174ccbd4a588cf7ffb1de42632e (DEFAULT_TABLE table=default.Table_48d4c9741adf4c52b6c0732c56987255 [id=cf499dac577641bd8cd74d615076204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.057762 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet f0287bc315644f0aaa976a56174576d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a27d21de6a6441d810f379bba88f9a2) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.062002 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:15.066402 28150 leader_election.cc:341] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.066555 28150 leader_election.cc:302] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:15.066669 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 7549b2ca31b04247bf3a2202e7585817 (DEFAULT_TABLE table=default.Table_f0b1e4dd193a4d6aa74cbc8280c025b7 [id=d1d16a2ea24843e59e608b2adc7f19a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.066834 29503 raft_consensus.cc:2726] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.067291 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 215a6f53e47e49cb984565adf659614d (DEFAULT_TABLE table=default.Table_b8a25247ce2a4633b8e8bc77291fb69d [id=a9f800d64ab54e0bb7c6a6e2601a290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.067216 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0b5dae96b3b64066ba7334dc73e578e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:15.067684 28604 tablet_bootstrap.cc:492] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.065703 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1d2243e88c784b929de2fc4d94804219" } new_table_name: "default.Table_12b9dd4031b04dccb4e6df3318d79588" modify_external_catalogs: true
I20240503 14:12:15.068688 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 57a959b76bc847ad82fbc847d9947c16 (DEFAULT_TABLE table=default.Table_1ddf62892e2149ccb722a65a606c43e7 [id=f1f757950d5f486bbc785c75e4fd1ef7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:15.069229 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b5dae96b3b64066ba7334dc73e578e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.070107 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 137c9eef465843049e93374367e562af (DEFAULT_TABLE table=default.Table_b509f93207e3417c9bfe50c97d92096e [id=ad58d0f68c6744a98f0b2c98fd56b26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.071113 28604 tablet_bootstrap.cc:654] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.074123 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7927f3e0fb214441aaf52352b6956ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.078155 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae05e174ccbd4a588cf7ffb1de42632e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.043610 28600 consensus_queue.cc:260] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.045387 29516 raft_consensus.cc:686] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:15.081642 28600 ts_tablet_manager.cc:1398] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.059s user 0.001s sys 0.001s
I20240503 14:12:15.081995 28600 tablet_bootstrap.cc:492] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.082849 28614 tablet_bootstrap.cc:492] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.083017 28614 ts_tablet_manager.cc:1367] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.235s user 0.047s sys 0.060s
I20240503 14:12:15.083765 28614 raft_consensus.cc:348] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.083956 28614 raft_consensus.cc:374] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.084002 28614 raft_consensus.cc:729] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.084230 28614 consensus_queue.cc:260] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.086162 28614 ts_tablet_manager.cc:1398] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:15.086530 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7549b2ca31b04247bf3a2202e7585817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.086635 28614 tablet_bootstrap.cc:492] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.087579 28185 tablet_service.cc:1467] Processing CreateTablet for tablet ae05e174ccbd4a588cf7ffb1de42632e (DEFAULT_TABLE table=default.Table_48d4c9741adf4c52b6c0732c56987255 [id=cf499dac577641bd8cd74d615076204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.087975 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae05e174ccbd4a588cf7ffb1de42632e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.091926 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215a6f53e47e49cb984565adf659614d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.095000 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a959b76bc847ad82fbc847d9947c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.095491 28661 ts_tablet_manager.cc:1888] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:15.081720 29516 consensus_queue.cc:237] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.095677 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_29f03216d162412bb8421c8c6fe741e4" } new_table_name: "default.Table_5354b912229e43539928c0e0df20a641" modify_external_catalogs: true
I20240503 14:12:15.095944 28661 log.cc:1196] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/651478ef632144079f93f3b50bff4611
I20240503 14:12:15.099804 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c9eef465843049e93374367e562af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.100203 28661 ts_tablet_manager.cc:1909] T 651478ef632144079f93f3b50bff4611 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:15.104738 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2a90167ae51647688dfc502b00c5c9c6
I20240503 14:12:15.105569 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 77f6fb009bdd4f70ba90e7b5f972cea9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:15.105777 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_222131e1d2a848758bd7e6b5a7498f34" } modify_external_catalogs: true
I20240503 14:12:15.105938 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_19f17d20edd24196b9efade6dca84369" } modify_external_catalogs: true
I20240503 14:12:15.106009 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_222131e1d2a848758bd7e6b5a7498f34" } modify_external_catalogs: true
I20240503 14:12:15.106182 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_19f17d20edd24196b9efade6dca84369" } modify_external_catalogs: true
I20240503 14:12:15.107046 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2cad43f3c6094253ba87d0ed3fc29fd9
I20240503 14:12:15.107561 28033 catalog_manager.cc:6554] ReplaceTablet: tablet 2cad43f3c6094253ba87d0ed3fc29fd9 deleted and replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57
W20240503 14:12:15.108062 28019 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet d832cd30f2414303818d34a0cdf47e60 consensus state has no leader
I20240503 14:12:15.109815 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a90167ae51647688dfc502b00c5c9c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.111363 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.112941 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_738df9de21164980b362ce19f71f3d9b" } modify_external_catalogs: true
I20240503 14:12:15.113543 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_738df9de21164980b362ce19f71f3d9b" } modify_external_catalogs: true
I20240503 14:12:15.114719 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b139ab0b5864024ac48215d46b0a7bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:15.116314 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5f6c2307b8e64c2481d5bd47e9189fea
I20240503 14:12:15.116813 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 137c9eef465843049e93374367e562af (DEFAULT_TABLE table=default.Table_b509f93207e3417c9bfe50c97d92096e [id=ad58d0f68c6744a98f0b2c98fd56b26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.116904 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 499) took 1819 ms. Trace:
I20240503 14:12:15.117285 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c9eef465843049e93374367e562af. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:15.117344 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07354861ee6142d6b1fbe573397d3160 consensus state has no leader
W20240503 14:12:15.117481 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2a90167ae51647688dfc502b00c5c9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a90167ae51647688dfc502b00c5c9c6 already in progress: creating tablet
W20240503 14:12:15.118425 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cad43f3c6094253ba87d0ed3fc29fd9 already in progress: creating tablet
I20240503 14:12:15.119910 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a90167ae51647688dfc502b00c5c9c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.120698 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fb7b6285a62e4ccaad9e7f7a3d82c1aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.121402 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb7b6285a62e4ccaad9e7f7a3d82c1aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.123174 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 57a959b76bc847ad82fbc847d9947c16 (DEFAULT_TABLE table=default.Table_1ddf62892e2149ccb722a65a606c43e7 [id=f1f757950d5f486bbc785c75e4fd1ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.123597 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a959b76bc847ad82fbc847d9947c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.123998 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:15.124975 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2f0f2612975448ecb7216f5e7b1dc2b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.125584 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f0f2612975448ecb7216f5e7b1dc2b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.126807 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 7927f3e0fb214441aaf52352b6956ef6 (DEFAULT_TABLE table=default.Table_dcf7f653c09f4112b516cee572ea9993 [id=eb7b7f9826ee4073969a65f9622567e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.127198 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7927f3e0fb214441aaf52352b6956ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.132547 28471 tablet_service.cc:1467] Processing CreateTablet for tablet ae05e174ccbd4a588cf7ffb1de42632e (DEFAULT_TABLE table=default.Table_48d4c9741adf4c52b6c0732c56987255 [id=cf499dac577641bd8cd74d615076204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.132841 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 7549b2ca31b04247bf3a2202e7585817 (DEFAULT_TABLE table=default.Table_f0b1e4dd193a4d6aa74cbc8280c025b7 [id=d1d16a2ea24843e59e608b2adc7f19a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.133030 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae05e174ccbd4a588cf7ffb1de42632e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.133167 28604 tablet_bootstrap.cc:492] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.133235 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7549b2ca31b04247bf3a2202e7585817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.131181 28661 rpcz_store.cc:276] 0503 14:12:13.282459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.282527 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:14.795645 (+1513118us) ts_tablet_manager.cc:1120] Deleting tablet 651478ef632144079f93f3b50bff4611
0503 14:12:14.795655 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:14.982899 (+187244us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.040006 (+ 57107us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.102164 (+ 62158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1512804}
I20240503 14:12:15.136337 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 215a6f53e47e49cb984565adf659614d (DEFAULT_TABLE table=default.Table_b8a25247ce2a4633b8e8bc77291fb69d [id=a9f800d64ab54e0bb7c6a6e2601a290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.136773 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215a6f53e47e49cb984565adf659614d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.151540 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6c2307b8e64c2481d5bd47e9189fea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.152603 28614 tablet_bootstrap.cc:654] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:15.152704 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5f6c2307b8e64c2481d5bd47e9189fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f6c2307b8e64c2481d5bd47e9189fea already in progress: creating tablet
I20240503 14:12:15.153963 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.163422 28614 tablet_bootstrap.cc:492] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.163555 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6c2307b8e64c2481d5bd47e9189fea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.163697 28614 ts_tablet_manager.cc:1367] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.000s
I20240503 14:12:15.164703 28614 raft_consensus.cc:348] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.165014 28614 raft_consensus.cc:374] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.165143 28614 raft_consensus.cc:729] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.165519 28614 consensus_queue.cc:260] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.166127 28614 ts_tablet_manager.cc:1398] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:15.166982 28614 tablet_bootstrap.cc:492] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.169934 28614 tablet_bootstrap.cc:654] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.175374 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b139ab0b5864024ac48215d46b0a7bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:15.175472 28032 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:15.175627 28214 raft_consensus.cc:3037] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.176409 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9e9d878dfb144e5db914f8619c13ce8c
I20240503 14:12:15.177286 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f40d0e0c2ffb4c3a8169abdfdc3f8148"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.178025 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f40d0e0c2ffb4c3a8169abdfdc3f8148 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.179841 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d3943c8358a3484d943bad0c906a9425
I20240503 14:12:15.181428 29503 raft_consensus.cc:482] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.181568 29503 raft_consensus.cc:504] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.182157 29503 leader_election.cc:288] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:15.183949 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9d878dfb144e5db914f8619c13ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.184203 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet d3943c8358a3484d943bad0c906a9425 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:15.186903 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d3943c8358a3484d943bad0c906a9425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3943c8358a3484d943bad0c906a9425 already in progress: creating tablet
I20240503 14:12:15.187274 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad1cd0e80cd4adbbbbcd60e335a5c49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:15.187562 28352 raft_consensus.cc:2454] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
W20240503 14:12:15.187619 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9e9d878dfb144e5db914f8619c13ce8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9d878dfb144e5db914f8619c13ce8c already in progress: creating tablet
I20240503 14:12:15.188751 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 77f6fb009bdd4f70ba90e7b5f972cea9
I20240503 14:12:15.189324 28027 catalog_manager.cc:6554] ReplaceTablet: tablet 77f6fb009bdd4f70ba90e7b5f972cea9 deleted and replaced by tablet 2946ba624b244db9a617f2e202bfc1ec
I20240503 14:12:15.190279 28150 leader_election.cc:302] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
W20240503 14:12:15.192157 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b24ab3a5bba04a5092225a981f292222 consensus state has no leader
I20240503 14:12:15.193236 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2946ba624b244db9a617f2e202bfc1ec) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:15.193990 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f6fb009bdd4f70ba90e7b5f972cea9 already in progress: creating tablet
I20240503 14:12:15.195287 28600 tablet_bootstrap.cc:654] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.195767 29503 raft_consensus.cc:2781] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.195847 29503 raft_consensus.cc:482] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.195909 29503 raft_consensus.cc:3037] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.195940 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9d878dfb144e5db914f8619c13ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.201059 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2946ba624b244db9a617f2e202bfc1ec) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.205132 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 1044f34588d045ce898435b61a90503f from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:15.205543 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 7927f3e0fb214441aaf52352b6956ef6 (DEFAULT_TABLE table=default.Table_dcf7f653c09f4112b516cee572ea9993 [id=eb7b7f9826ee4073969a65f9622567e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.206017 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7927f3e0fb214441aaf52352b6956ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.206931 29429 raft_consensus.cc:482] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.207854 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet d3943c8358a3484d943bad0c906a9425 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.209975 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 with delete_type TABLET_DATA_DELETED (replaced by tablet 752b369f24224f2486834814a9d08495) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.210253 28603 tablet_replica.cc:330] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:15.210465 28603 raft_consensus.cc:2229] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.210657 28603 raft_consensus.cc:2258] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.210813 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 500) took 1666 ms. Trace:
I20240503 14:12:15.210942 28661 rpcz_store.cc:276] 0503 14:12:13.468606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.469780 (+ 1174us) service_pool.cc:225] Handling call
0503 14:12:15.135372 (+1665592us) ts_tablet_manager.cc:1120] Deleting tablet a6c17cfce7e644668817bf3483a098d4
0503 14:12:15.135382 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.135515 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665250}
I20240503 14:12:15.212507 28603 ts_tablet_manager.cc:1875] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.213049 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 508) took 1609 ms. Trace:
I20240503 14:12:15.213181 28661 rpcz_store.cc:276] 0503 14:12:13.601856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.601937 (+ 81us) service_pool.cc:225] Handling call
0503 14:12:15.211223 (+1609286us) ts_tablet_manager.cc:1120] Deleting tablet 34fa067359a640eeba8c80d08371904e
0503 14:12:15.211234 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.211359 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608132}
I20240503 14:12:15.213783 28604 ts_tablet_manager.cc:1367] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.008s
I20240503 14:12:15.149945 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 7549b2ca31b04247bf3a2202e7585817 (DEFAULT_TABLE table=default.Table_f0b1e4dd193a4d6aa74cbc8280c025b7 [id=d1d16a2ea24843e59e608b2adc7f19a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.213986 29480 consensus_queue.cc:1035] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.214258 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 215a6f53e47e49cb984565adf659614d (DEFAULT_TABLE table=default.Table_b8a25247ce2a4633b8e8bc77291fb69d [id=a9f800d64ab54e0bb7c6a6e2601a290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.214622 28661 tablet_replica.cc:330] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:15.214840 28661 raft_consensus.cc:2229] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.215054 28661 raft_consensus.cc:2258] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.215593 28661 ts_tablet_manager.cc:1875] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:15.215723 28150 consensus_peers.cc:485] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:15.221032 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad1cd0e80cd4adbbbbcd60e335a5c49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:15.223184 29429 raft_consensus.cc:504] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.223378 29503 raft_consensus.cc:504] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.224017 29429 leader_election.cc:288] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:15.224555 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad1cd0e80cd4adbbbbcd60e335a5c49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:15.224685 28150 leader_election.cc:341] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.225366 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215a6f53e47e49cb984565adf659614d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.225715 29503 leader_election.cc:288] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:15.226928 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_29aaa3a8d8ed47dab6dfc37b3989f427"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.227602 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29aaa3a8d8ed47dab6dfc37b3989f427 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.228212 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 137c9eef465843049e93374367e562af (DEFAULT_TABLE table=default.Table_b509f93207e3417c9bfe50c97d92096e [id=ad58d0f68c6744a98f0b2c98fd56b26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.228654 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c9eef465843049e93374367e562af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.229593 28214 raft_consensus.cc:2454] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:15.232209 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a90167ae51647688dfc502b00c5c9c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.232486 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 57a959b76bc847ad82fbc847d9947c16 (DEFAULT_TABLE table=default.Table_1ddf62892e2149ccb722a65a606c43e7 [id=f1f757950d5f486bbc785c75e4fd1ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.233187 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.234302 28604 raft_consensus.cc:348] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.235103 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a959b76bc847ad82fbc847d9947c16. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:15.237458 28150 leader_election.cc:341] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:15.251164 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad1cd0e80cd4adbbbbcd60e335a5c49" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:15.251994 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9d878dfb144e5db914f8619c13ce8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.252291 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6c2307b8e64c2481d5bd47e9189fea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.252494 28604 raft_consensus.cc:374] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.254523 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_85c71baa763440bf939ea1193fa0af0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.254699 28150 leader_election.cc:341] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.254822 28150 leader_election.cc:302] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:15.255257 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85c71baa763440bf939ea1193fa0af0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:15.255618 28427 leader_election.cc:341] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.256763 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f6fdd56971514ba4a525832dbcd2bbf7" } new_table_name: "default.Table_e65026502109470c9382acbdf03874d6" modify_external_catalogs: true
I20240503 14:12:15.258927 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7549b2ca31b04247bf3a2202e7585817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.259020 29503 raft_consensus.cc:2726] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.259104 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e27287910d04e1b9c08ae3eca4739b0" } new_table_name: "default.Table_cfcbfe39af47450d9a0917259530eb81" modify_external_catalogs: true
W20240503 14:12:15.255965 28427 consensus_peers.cc:485] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c7d57fc4486457e9848daad79f04356. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:15.259878 28604 raft_consensus.cc:729] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.260313 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2946ba624b244db9a617f2e202bfc1ec) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.260372 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet d3943c8358a3484d943bad0c906a9425 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.262653 28603 ts_tablet_manager.cc:1888] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.262897 28603 log.cc:1196] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0ad1cd0e80cd4adbbbbcd60e335a5c49
I20240503 14:12:15.263221 28603 ts_tablet_manager.cc:1909] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:15.264375 28427 consensus_peers.cc:485] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:15.267434 28427 leader_election.cc:302] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.268774 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (table default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]) successfully deleted
I20240503 14:12:15.269279 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]) successfully deleted
W20240503 14:12:15.269551 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a6c17cfce7e644668817bf3483a098d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c17cfce7e644668817bf3483a098d4 already in progress: creating tablet
W20240503 14:12:15.269879 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 34fa067359a640eeba8c80d08371904e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34fa067359a640eeba8c80d08371904e already in progress: creating tablet
I20240503 14:12:15.271173 29429 raft_consensus.cc:2781] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.271348 29429 raft_consensus.cc:686] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:15.276875 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet cc230b489ab8455c8cb2eba31084fddb
I20240503 14:12:15.277253 28604 consensus_queue.cc:260] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.277572 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet aa3ce05b422a4166bfac468acb84e4cc from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:15.278790 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet cc230b489ab8455c8cb2eba31084fddb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:15.278853 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99aa7e59c45c45c7bc1f726c97426b89 consensus state has no leader
W20240503 14:12:15.279721 28427 consensus_peers.cc:485] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:15.279973 28427 leader_election.cc:341] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0bd5234791524eddb7edd493e39fb993
I20240503 14:12:15.280004 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet cc230b489ab8455c8cb2eba31084fddb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.280311 28603 tablet_replica.cc:330] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:15.280498 28603 raft_consensus.cc:2229] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.280766 28603 raft_consensus.cc:2258] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.281085 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet cc230b489ab8455c8cb2eba31084fddb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.275691 29429 consensus_queue.cc:237] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.281412 28603 ts_tablet_manager.cc:1875] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.285507 28604 ts_tablet_manager.cc:1398] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.070s user 0.001s sys 0.002s
I20240503 14:12:15.289665 28653 ts_tablet_manager.cc:1888] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.290041 28604 tablet_bootstrap.cc:492] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.292371 28653 log.cc:1196] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/3703ab7db61b4c9ead3058bfdc123189
W20240503 14:12:15.281117 28427 leader_election.cc:341] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 0bd5234791524eddb7edd493e39fb993
I20240503 14:12:15.293036 28653 ts_tablet_manager.cc:1909] T 3703ab7db61b4c9ead3058bfdc123189 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.294931 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 435) took 3253 ms. Trace:
I20240503 14:12:15.301118 28427 leader_election.cc:302] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.296002 28600 tablet_bootstrap.cc:492] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.301573 28600 ts_tablet_manager.cc:1367] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.220s user 0.004s sys 0.008s
I20240503 14:12:15.302561 28600 raft_consensus.cc:348] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.302878 28600 raft_consensus.cc:374] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.303107 28600 raft_consensus.cc:729] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.303427 28600 consensus_queue.cc:260] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.304016 28600 ts_tablet_manager.cc:1398] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:15.305194 28600 tablet_bootstrap.cc:492] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.305778 28604 tablet_bootstrap.cc:654] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.308813 29515 raft_consensus.cc:2726] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.310276 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]). Expected version 2 got 0
I20240503 14:12:15.310472 28661 ts_tablet_manager.cc:1888] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:15.312964 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d43823da09bb49be8f5d1a1149a4b11d consensus state has no leader
W20240503 14:12:15.313519 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2946ba624b244db9a617f2e202bfc1ec has no consensus state
I20240503 14:12:15.315865 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c12ba1b49a3f4f509ff427bbeb4723af" } modify_external_catalogs: true
I20240503 14:12:15.316112 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c12ba1b49a3f4f509ff427bbeb4723af" } modify_external_catalogs: true
I20240503 14:12:15.317803 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1df0f31c455445fbac525618cff09685"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.318395 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1df0f31c455445fbac525618cff09685 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.319224 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f0b1e4dd193a4d6aa74cbc8280c025b7" } new_table_name: "default.Table_4489c34fb7d34e72892ce61b61a5fe17" modify_external_catalogs: true
I20240503 14:12:15.321179 28614 tablet_bootstrap.cc:492] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.321337 28614 ts_tablet_manager.cc:1367] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.154s user 0.008s sys 0.001s
I20240503 14:12:15.322253 28614 raft_consensus.cc:348] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.322464 28614 raft_consensus.cc:374] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.322525 28614 raft_consensus.cc:729] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.321472 28653 rpcz_store.cc:276] 0503 14:12:12.041077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.041420 (+ 343us) service_pool.cc:225] Handling call
0503 14:12:15.020340 (+2978920us) ts_tablet_manager.cc:1120] Deleting tablet 3703ab7db61b4c9ead3058bfdc123189
0503 14:12:15.020351 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.245636 (+225285us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.289536 (+ 43900us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.294906 (+ 5370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2978564}
I20240503 14:12:15.322497 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_34e94ee61c2644d3b8f4449f0d6f341b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:15.322793 28653 tablet_replica.cc:330] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.322722 28614 consensus_queue.cc:260] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:15.323068 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34e94ee61c2644d3b8f4449f0d6f341b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.323218 28614 ts_tablet_manager.cc:1398] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:15.323477 28614 tablet_bootstrap.cc:492] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.324196 28600 tablet_bootstrap.cc:654] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.326164 28653 raft_consensus.cc:2229] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.326344 28653 raft_consensus.cc:2258] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.325162 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_960c4947e6bc4bb6839d7257e7d93a41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.329607 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_960c4947e6bc4bb6839d7257e7d93a41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.331719 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e65026502109470c9382acbdf03874d6" } modify_external_catalogs: true
I20240503 14:12:15.331979 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e65026502109470c9382acbdf03874d6" } modify_external_catalogs: true
I20240503 14:12:15.332161 28653 ts_tablet_manager.cc:1875] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.335156 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb (DEFAULT_TABLE table=default.Table_ec78ac6bb77c4e4790e3d2c0f107c6e7 [id=7e2c5447ab2e4faaafde4e78d940c795]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.335604 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.336347 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 1b523d49122b4b279cbfe7615aaaabc2 (DEFAULT_TABLE table=default.Table_62b7129591554425bc6c23af414fbffb [id=6e62a3c0104f4956b09847dfa74baf89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.336768 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b523d49122b4b279cbfe7615aaaabc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.340435 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 1b523d49122b4b279cbfe7615aaaabc2 (DEFAULT_TABLE table=default.Table_62b7129591554425bc6c23af414fbffb [id=6e62a3c0104f4956b09847dfa74baf89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.340842 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b523d49122b4b279cbfe7615aaaabc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.341141 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 6a27d21de6a6441d810f379bba88f9a2 (DEFAULT_TABLE table=default.Table_a28d952dbd9141a9a113c42bcbc27a14 [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.341485 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a27d21de6a6441d810f379bba88f9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.342550 28614 tablet_bootstrap.cc:654] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.345369 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 6a27d21de6a6441d810f379bba88f9a2 (DEFAULT_TABLE table=default.Table_a28d952dbd9141a9a113c42bcbc27a14 [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.345736 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a27d21de6a6441d810f379bba88f9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.349164 29281 raft_consensus.cc:482] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.349363 29281 raft_consensus.cc:504] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.350754 29281 leader_election.cc:288] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:15.350832 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:15.351397 28150 leader_election.cc:341] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.356792 28653 ts_tablet_manager.cc:1888] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.356942 28653 log.cc:1196] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0b139ab0b5864024ac48215d46b0a7bd
I20240503 14:12:15.357256 28653 ts_tablet_manager.cc:1909] T 0b139ab0b5864024ac48215d46b0a7bd P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.364548 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 47965a978a904f2d876690de01f89771 (DEFAULT_TABLE table=default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.364548 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 47965a978a904f2d876690de01f89771 (DEFAULT_TABLE table=default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.365008 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47965a978a904f2d876690de01f89771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.365008 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47965a978a904f2d876690de01f89771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.367995 28603 ts_tablet_manager.cc:1888] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:15.368142 28603 log.cc:1196] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/cc230b489ab8455c8cb2eba31084fddb
I20240503 14:12:15.368497 28603 ts_tablet_manager.cc:1909] T cc230b489ab8455c8cb2eba31084fddb P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:15.369244 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 1292a339f00247369ebeaa35752d51f8 (DEFAULT_TABLE table=default.Table_84ebaac5471744d0ad8b983a03565cee [id=4bc361205572492d968f4ee9841ae645]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.369689 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1292a339f00247369ebeaa35752d51f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.369649 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f08aac9a7147ebaffda4262a57a86a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.369885 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet e135b60660f448a7974520f2f87cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.369997 28603 tablet_replica.cc:330] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:15.370175 28603 raft_consensus.cc:2229] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.370337 28603 raft_consensus.cc:2258] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.370368 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 1292a339f00247369ebeaa35752d51f8 (DEFAULT_TABLE table=default.Table_84ebaac5471744d0ad8b983a03565cee [id=4bc361205572492d968f4ee9841ae645]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.370776 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1292a339f00247369ebeaa35752d51f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.371119 28603 ts_tablet_manager.cc:1875] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.373124 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 1132014802ed4c3682769ba86312eed2 (DEFAULT_TABLE table=default.Table_9d3155335c5a46209c736c3c3ccd5eb4 [id=78addffa960c483c91ada5b23e093086]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.373515 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1132014802ed4c3682769ba86312eed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.376912 28614 tablet_bootstrap.cc:492] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.377050 28614 ts_tablet_manager.cc:1367] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240503 14:12:15.377811 28614 raft_consensus.cc:348] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.378018 28614 raft_consensus.cc:374] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.378077 28614 raft_consensus.cc:729] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.378304 28614 consensus_queue.cc:260] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.378855 28614 ts_tablet_manager.cc:1398] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:15.379158 28614 tablet_bootstrap.cc:492] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.385169 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cf24dd3520bd4373acfd719f2c7737ad" } modify_external_catalogs: true
I20240503 14:12:15.385459 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cf24dd3520bd4373acfd719f2c7737ad" } modify_external_catalogs: true
I20240503 14:12:15.386394 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f2b1cc65e0f5452fbc24389d16be4eb5" } modify_external_catalogs: true
I20240503 14:12:15.386653 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f2b1cc65e0f5452fbc24389d16be4eb5" } modify_external_catalogs: true
I20240503 14:12:15.386576 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 436) took 3344 ms. Trace:
I20240503 14:12:15.386953 28653 rpcz_store.cc:276] 0503 14:12:12.042103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.042160 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:15.322735 (+3280575us) ts_tablet_manager.cc:1120] Deleting tablet 0b139ab0b5864024ac48215d46b0a7bd
0503 14:12:15.322747 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.353692 (+ 30945us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.356688 (+ 2996us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.386545 (+ 29857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3280325}
I20240503 14:12:15.387421 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_75a7cf3d16ac48e69b1ee39530fedaa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:15.387960 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 437) took 3299 ms. Trace:
W20240503 14:12:15.388049 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75a7cf3d16ac48e69b1ee39530fedaa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.388041 28653 rpcz_store.cc:276] 0503 14:12:12.088616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.088681 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:15.387786 (+3299105us) ts_tablet_manager.cc:1120] Deleting tablet ff4d28d3c2ea472191ad21355248a2fc
0503 14:12:15.387798 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.387940 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3291830}
I20240503 14:12:15.391476 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 3703ab7db61b4c9ead3058bfdc123189 (table default.Table_fb279b38e7764d3bac7f2aa1667ca58e [id=19a9b36e090c46a483c355f0f5cd6c43]) successfully deleted
I20240503 14:12:15.392052 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0b139ab0b5864024ac48215d46b0a7bd (table default.Table_52932a5c26744566bfc744e2ed1d8aee [id=5c194c0fc1c94da0a40f0f3f0d7484b7]) successfully deleted
W20240503 14:12:15.392498 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ff4d28d3c2ea472191ad21355248a2fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4d28d3c2ea472191ad21355248a2fc already in progress: creating tablet
I20240503 14:12:15.394269 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_530c3b5891cd412d8b15b2c2b91de623" } new_table_name: "default.Table_2ce3b6f2b77b4607856fc2a64ead5bb5" modify_external_catalogs: true
I20240503 14:12:15.394393 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 37dac592c9dd4888a2ce89fa922f7fc8 (DEFAULT_TABLE table=default.Table_0d81fa49986c4ca99528a823b4a22ae6 [id=312dbb7ac9cd4f32b4b8593adf25f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.394836 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dac592c9dd4888a2ce89fa922f7fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.395184 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 438) took 3305 ms. Trace:
I20240503 14:12:15.395278 28653 rpcz_store.cc:276] 0503 14:12:12.089526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.089588 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:15.395026 (+3305438us) ts_tablet_manager.cc:1120] Deleting tablet 9ac2833a0bb045dcb2824468d936ffe4
0503 14:12:15.395035 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.395160 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3298999}
I20240503 14:12:15.395591 28653 tablet_replica.cc:330] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.395773 28653 raft_consensus.cc:2229] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.395925 28653 raft_consensus.cc:2258] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.396924 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 37dac592c9dd4888a2ce89fa922f7fc8 (DEFAULT_TABLE table=default.Table_0d81fa49986c4ca99528a823b4a22ae6 [id=312dbb7ac9cd4f32b4b8593adf25f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.397415 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dac592c9dd4888a2ce89fa922f7fc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:15.401841 28150 consensus_peers.cc:485] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc230b489ab8455c8cb2eba31084fddb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:15.402547 28653 ts_tablet_manager.cc:1875] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.404000 28614 tablet_bootstrap.cc:654] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.405608 28603 ts_tablet_manager.cc:1888] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.405718 28603 log.cc:1196] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/54f08aac9a7147ebaffda4262a57a86a
I20240503 14:12:15.405988 28603 ts_tablet_manager.cc:1909] T 54f08aac9a7147ebaffda4262a57a86a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:15.406579 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 1132014802ed4c3682769ba86312eed2 (DEFAULT_TABLE table=default.Table_9d3155335c5a46209c736c3c3ccd5eb4 [id=78addffa960c483c91ada5b23e093086]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.406962 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1132014802ed4c3682769ba86312eed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.412724 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.413089 28603 tablet_replica.cc:330] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:15.413053 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76]) successfully deleted
I20240503 14:12:15.413327 28603 raft_consensus.cc:2229] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.413527 28603 raft_consensus.cc:2258] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.413991 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 54f08aac9a7147ebaffda4262a57a86a (table default.Table_fc0d4a5f37ae46d9807079682b94ba1d [id=f406220394264c8f824c0e1fce8479e1]) successfully deleted
I20240503 14:12:15.414242 28603 ts_tablet_manager.cc:1875] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:15.414474 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9ac2833a0bb045dcb2824468d936ffe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ac2833a0bb045dcb2824468d936ffe4 already in progress: creating tablet
I20240503 14:12:15.417838 28661 log.cc:1196] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/3514b79467dc42bc89fdacf8e39a000d
I20240503 14:12:15.420043 28614 tablet_bootstrap.cc:492] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.420296 28614 ts_tablet_manager.cc:1367] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20240503 14:12:15.421229 28614 raft_consensus.cc:348] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.421551 28614 raft_consensus.cc:374] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.421644 28614 raft_consensus.cc:729] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.421911 28614 consensus_queue.cc:260] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.423274 28614 ts_tablet_manager.cc:1398] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:15.424885 28603 ts_tablet_manager.cc:1888] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.425067 28603 log.cc:1196] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/50edc38cdbe74397aa4f4928d56d9fbc
I20240503 14:12:15.425431 28603 ts_tablet_manager.cc:1909] T 50edc38cdbe74397aa4f4928d56d9fbc P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:15.426196 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb (DEFAULT_TABLE table=default.Table_ec78ac6bb77c4e4790e3d2c0f107c6e7 [id=7e2c5447ab2e4faaafde4e78d940c795]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.426928 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.427242 28653 ts_tablet_manager.cc:1888] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:15.427484 28653 log.cc:1196] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0f4fdf88fefa4ff7939f3ca05a82d7ba
I20240503 14:12:15.428496 28653 ts_tablet_manager.cc:1909] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.430003 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 452) took 3099 ms. Trace:
I20240503 14:12:15.431960 28653 rpcz_store.cc:276] 0503 14:12:12.331045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.331152 (+ 107us) service_pool.cc:225] Handling call
0503 14:12:15.395546 (+3064394us) ts_tablet_manager.cc:1120] Deleting tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba
0503 14:12:15.395555 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.405881 (+ 10326us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.425656 (+ 19775us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.429977 (+ 4321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3064069}
I20240503 14:12:15.432363 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:15.434531 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7549b2ca31b04247bf3a2202e7585817 consensus state has no leader
I20240503 14:12:15.434789 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 79d09bee02c14822aa08050eac5be181
I20240503 14:12:15.437320 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 5a081ba59ad44890bf2f94874c034277 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:15.438408 28661 ts_tablet_manager.cc:1909] T 3514b79467dc42bc89fdacf8e39a000d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:15.438848 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 509) took 1835 ms. Trace:
I20240503 14:12:15.439203 28661 rpcz_store.cc:276] 0503 14:12:13.603214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.603527 (+ 313us) service_pool.cc:225] Handling call
0503 14:12:15.213444 (+1609917us) ts_tablet_manager.cc:1120] Deleting tablet 3514b79467dc42bc89fdacf8e39a000d
0503 14:12:15.213452 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.219193 (+ 5741us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.298018 (+ 78825us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.438801 (+140783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601792}
I20240503 14:12:15.439760 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 510) took 1815 ms. Trace:
I20240503 14:12:15.440105 28661 rpcz_store.cc:276] 0503 14:12:13.623848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.623911 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:15.439520 (+1815609us) ts_tablet_manager.cc:1120] Deleting tablet 60206e8e73e84e409ed6aba222d9ff4b
0503 14:12:15.439530 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.439700 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815273}
I20240503 14:12:15.440550 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 519) took 1615 ms. Trace:
I20240503 14:12:15.440858 28661 rpcz_store.cc:276] 0503 14:12:13.825451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.825649 (+ 198us) service_pool.cc:225] Handling call
0503 14:12:15.440407 (+1614758us) ts_tablet_manager.cc:1120] Deleting tablet ac752d8ab4fd4a218b052f388f416494
0503 14:12:15.440416 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.440498 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608139}
I20240503 14:12:15.441179 28661 tablet_replica.cc:330] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:15.441395 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d43823da09bb49be8f5d1a1149a4b11d
I20240503 14:12:15.441635 28661 raft_consensus.cc:2229] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.441843 28661 raft_consensus.cc:2258] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:15.434161 28150 leader_election.cc:341] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.442474 28150 leader_election.cc:302] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:15.442664 28661 ts_tablet_manager.cc:1875] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.442971 29281 raft_consensus.cc:2726] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.437884 28600 tablet_bootstrap.cc:492] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.443234 28600 ts_tablet_manager.cc:1367] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.138s user 0.009s sys 0.009s
I20240503 14:12:15.444058 28600 raft_consensus.cc:348] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.444344 28600 raft_consensus.cc:374] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.444445 28600 raft_consensus.cc:729] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.444713 28600 consensus_queue.cc:260] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.450345 28600 ts_tablet_manager.cc:1398] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:12:15.455572 28600 tablet_bootstrap.cc:492] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.456607 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d09bee02c14822aa08050eac5be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.457759 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet d43823da09bb49be8f5d1a1149a4b11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.462141 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet c9325d5142af4b018354bf38370b61e0 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:15.464264 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d09bee02c14822aa08050eac5be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.466773 28653 tablet_replica.cc:330] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.467079 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba (table default.Table_a888de5f6a40467db886c3141d2a87cc [id=caca4dbaa701453794ac56475f46704e]) successfully deleted
I20240503 14:12:15.469652 28614 tablet_bootstrap.cc:492] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.471328 28614 tablet_bootstrap.cc:654] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.471648 28653 raft_consensus.cc:2229] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.472229 28653 raft_consensus.cc:2258] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.473138 28653 ts_tablet_manager.cc:1875] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.473297 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.473925 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet d43823da09bb49be8f5d1a1149a4b11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:15.477526 28661 ts_tablet_manager.cc:1888] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.469075 28600 tablet_bootstrap.cc:654] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.486282 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2ef0e74fb73844b3a4d3b680c04d73e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:15.486714 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_edb926a0a57b4cb090017b29fee48eab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.487144 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ef0e74fb73844b3a4d3b680c04d73e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:15.487413 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edb926a0a57b4cb090017b29fee48eab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.492511 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_90131726ecc0494f8e4eff1c20c6095f" } new_table_name: "default.Table_afadb9c7ad4a4debb06a243990cbc46e" modify_external_catalogs: true
I20240503 14:12:15.497294 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_618386316f9b46f9a7770c3f998769f5" } modify_external_catalogs: true
I20240503 14:12:15.497606 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_618386316f9b46f9a7770c3f998769f5" } modify_external_catalogs: true
I20240503 14:12:15.507066 28653 ts_tablet_manager.cc:1888] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.508539 28653 log.cc:1196] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/54f08aac9a7147ebaffda4262a57a86a
I20240503 14:12:15.511852 28653 ts_tablet_manager.cc:1909] T 54f08aac9a7147ebaffda4262a57a86a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.512480 29281 raft_consensus.cc:482] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.512634 29281 raft_consensus.cc:504] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.513444 29281 leader_election.cc:288] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:15.514533 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135b60660f448a7974520f2f87cf93a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:15.514966 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b139ab0b5864024ac48215d46b0a7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:15.515239 28150 leader_election.cc:341] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.515095 28600 tablet_bootstrap.cc:492] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.516212 28600 ts_tablet_manager.cc:1367] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.001s
W20240503 14:12:15.517160 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9325d5142af4b018354bf38370b61e0 consensus state has no leader
I20240503 14:12:15.517205 28600 raft_consensus.cc:348] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.517581 28600 raft_consensus.cc:374] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.517609 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 448) took 3212 ms. Trace:
I20240503 14:12:15.517710 28600 raft_consensus.cc:729] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.517799 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cc6397f0878d4dcb939227f869da0ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:15.517777 28653 rpcz_store.cc:276] 0503 14:12:12.305457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.331455 (+ 25998us) service_pool.cc:225] Handling call
0503 14:12:15.466709 (+3135254us) ts_tablet_manager.cc:1120] Deleting tablet 54f08aac9a7147ebaffda4262a57a86a
0503 14:12:15.466723 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.477370 (+ 10647us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.506936 (+ 29566us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.517589 (+ 10653us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4842,"tablet-delete.queue_time_us":3134912}
I20240503 14:12:15.518020 28600 consensus_queue.cc:260] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:15.518512 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc6397f0878d4dcb939227f869da0ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.518748 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 449) took 3213 ms. Trace:
I20240503 14:12:15.518860 28653 rpcz_store.cc:276] 0503 14:12:12.305577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.331849 (+ 26272us) service_pool.cc:225] Handling call
0503 14:12:15.518504 (+3186655us) ts_tablet_manager.cc:1120] Deleting tablet d7c79ffee3954b5c870f4a8aa6379517
0503 14:12:15.518543 (+ 39us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.518728 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8576,"tablet-delete.queue_time_us":3186419}
I20240503 14:12:15.519520 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 450) took 3213 ms. Trace:
I20240503 14:12:15.519615 28653 rpcz_store.cc:276] 0503 14:12:12.305724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.332145 (+ 26421us) service_pool.cc:225] Handling call
0503 14:12:15.519395 (+3187250us) ts_tablet_manager.cc:1120] Deleting tablet 0af0342e0e0b47499a9322b5b2cb3d03
0503 14:12:15.519404 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.519503 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3186981}
I20240503 14:12:15.520022 28653 tablet_replica.cc:330] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.520262 28653 raft_consensus.cc:2229] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.520484 28653 raft_consensus.cc:2258] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.521150 28653 ts_tablet_manager.cc:1875] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.522814 28600 ts_tablet_manager.cc:1398] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:15.523897 28600 tablet_bootstrap.cc:492] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.525943 28614 tablet_bootstrap.cc:492] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.526188 28614 ts_tablet_manager.cc:1367] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20240503 14:12:15.527184 28614 raft_consensus.cc:348] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.527478 28614 raft_consensus.cc:374] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.527585 28614 raft_consensus.cc:729] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.527892 28614 consensus_queue.cc:260] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.528486 28600 tablet_bootstrap.cc:654] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.530077 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 50edc38cdbe74397aa4f4928d56d9fbc (table default.Table_9a954e776a184d819708a62a99ed94fc [id=713e339595f142ab8a82bb724d23a487]) successfully deleted
I20240503 14:12:15.530452 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 54f08aac9a7147ebaffda4262a57a86a (table default.Table_fc0d4a5f37ae46d9807079682b94ba1d [id=f406220394264c8f824c0e1fce8479e1]) successfully deleted
W20240503 14:12:15.530977 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0af0342e0e0b47499a9322b5b2cb3d03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0af0342e0e0b47499a9322b5b2cb3d03 already in progress: creating tablet
I20240503 14:12:15.531010 28614 ts_tablet_manager.cc:1398] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20240503 14:12:15.531302 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 79d09bee02c14822aa08050eac5be181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79d09bee02c14822aa08050eac5be181 already in progress: creating tablet
I20240503 14:12:15.533813 28603 tablet_replica.cc:330] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:15.534075 28603 raft_consensus.cc:2229] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.534281 28603 raft_consensus.cc:2258] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.535800 28603 ts_tablet_manager.cc:1875] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.537053 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]). Expected version 2 got 0
I20240503 14:12:15.537279 28614 tablet_bootstrap.cc:492] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:15.538599 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 161008us
I20240503 14:12:15.538874 28604 tablet_bootstrap.cc:492] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.539101 28604 ts_tablet_manager.cc:1367] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.253s user 0.016s sys 0.021s
I20240503 14:12:15.540242 28604 raft_consensus.cc:348] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.540522 28604 raft_consensus.cc:374] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.540459 29516 consensus_queue.cc:1035] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:15.540661 28604 raft_consensus.cc:729] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.540971 29516 raft_consensus.cc:482] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.541136 29515 raft_consensus.cc:482] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.541307 29515 raft_consensus.cc:504] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.542434 29515 leader_election.cc:288] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:15.542609 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 1b523d49122b4b279cbfe7615aaaabc2 (DEFAULT_TABLE table=default.Table_62b7129591554425bc6c23af414fbffb [id=6e62a3c0104f4956b09847dfa74baf89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.542974 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb (DEFAULT_TABLE table=default.Table_ec78ac6bb77c4e4790e3d2c0f107c6e7 [id=7e2c5447ab2e4faaafde4e78d940c795]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.543157 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b523d49122b4b279cbfe7615aaaabc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.543309 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:15.543435 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.543529 28215 raft_consensus.cc:2454] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:15.543737 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 1292a339f00247369ebeaa35752d51f8 (DEFAULT_TABLE table=default.Table_84ebaac5471744d0ad8b983a03565cee [id=4bc361205572492d968f4ee9841ae645]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.543839 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 47965a978a904f2d876690de01f89771 (DEFAULT_TABLE table=default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.544294 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d09bee02c14822aa08050eac5be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.544387 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1292a339f00247369ebeaa35752d51f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.544898 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 1132014802ed4c3682769ba86312eed2 (DEFAULT_TABLE table=default.Table_9d3155335c5a46209c736c3c3ccd5eb4 [id=78addffa960c483c91ada5b23e093086]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.545639 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1132014802ed4c3682769ba86312eed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.548413 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47965a978a904f2d876690de01f89771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.548494 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 6a27d21de6a6441d810f379bba88f9a2 (DEFAULT_TABLE table=default.Table_a28d952dbd9141a9a113c42bcbc27a14 [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.549250 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:15.549634 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.549894 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9262a7e828489ab51a745068c737df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.550019 28427 leader_election.cc:302] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:15.541142 29516 raft_consensus.cc:504] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.550937 28653 ts_tablet_manager.cc:1888] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:15.550938 28427 leader_election.cc:341] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.550990 29429 raft_consensus.cc:2781] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.551059 28653 log.cc:1196] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/605531ed29214b1ca27011c5f5e96b11
I20240503 14:12:15.551124 29429 raft_consensus.cc:482] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.551234 29429 raft_consensus.cc:3037] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.551364 28653 ts_tablet_manager.cc:1909] T 605531ed29214b1ca27011c5f5e96b11 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.551779 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dbb59d98834ab6bbac4abfa92681e3" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:15.552193 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135b60660f448a7974520f2f87cf93a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:15.552484 28600 tablet_bootstrap.cc:492] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.552503 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet d43823da09bb49be8f5d1a1149a4b11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.552634 28600 ts_tablet_manager.cc:1367] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
W20240503 14:12:15.552736 28427 leader_election.cc:341] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:15.552768 28150 leader_election.cc:341] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.552874 28150 leader_election.cc:302] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:15.553030 28427 consensus_peers.cc:485] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54f08aac9a7147ebaffda4262a57a86a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:15.553277 29281 raft_consensus.cc:2726] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.553552 28600 raft_consensus.cc:348] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.553733 28600 raft_consensus.cc:374] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.553782 28600 raft_consensus.cc:729] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.553880 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 37dac592c9dd4888a2ce89fa922f7fc8 (DEFAULT_TABLE table=default.Table_0d81fa49986c4ca99528a823b4a22ae6 [id=312dbb7ac9cd4f32b4b8593adf25f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.553982 28600 consensus_queue.cc:260] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.554390 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dac592c9dd4888a2ce89fa922f7fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.554600 28600 ts_tablet_manager.cc:1398] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:15.554921 28600 tablet_bootstrap.cc:492] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.557230 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 3514b79467dc42bc89fdacf8e39a000d (table default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]) successfully deleted
W20240503 14:12:15.557623 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 60206e8e73e84e409ed6aba222d9ff4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60206e8e73e84e409ed6aba222d9ff4b already in progress: creating tablet
W20240503 14:12:15.557888 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ac752d8ab4fd4a218b052f388f416494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac752d8ab4fd4a218b052f388f416494 already in progress: creating tablet
I20240503 14:12:15.558204 29429 raft_consensus.cc:504] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.558557 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dbb59d98834ab6bbac4abfa92681e3" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:15.558763 28212 raft_consensus.cc:2454] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:15.559110 29429 leader_election.cc:288] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:15.559562 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:15.560406 28427 leader_election.cc:341] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.560642 28427 leader_election.cc:302] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.561750 28600 tablet_bootstrap.cc:654] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.562394 29516 leader_election.cc:288] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:15.562754 29516 raft_consensus.cc:2781] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.562829 29516 raft_consensus.cc:482] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.562881 29516 raft_consensus.cc:3037] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.563333 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a27d21de6a6441d810f379bba88f9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.565591 29516 raft_consensus.cc:504] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.540969 28604 consensus_queue.cc:260] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.566810 28604 ts_tablet_manager.cc:1398] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 14:12:15.567137 28604 tablet_bootstrap.cc:492] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.567373 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:15.567579 28212 raft_consensus.cc:3037] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.569193 28604 tablet_bootstrap.cc:654] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:15.574846 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cace51f8e5ff49f596c94f3698fdf9ef consensus state has no leader
I20240503 14:12:15.575978 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_11a3df4a63e54fa98b91c2a818269000" } new_table_name: "default.Table_13b6a3c3a46b4308b4ce309165333325" modify_external_catalogs: true
I20240503 14:12:15.577311 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f9806bd6f25a48bfb0a4a65437072534"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.577966 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9806bd6f25a48bfb0a4a65437072534 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.578361 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8796e63f1b454ff08f86549980fc079d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.578981 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8796e63f1b454ff08f86549980fc079d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.579447 28600 tablet_bootstrap.cc:492] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.579612 28600 ts_tablet_manager.cc:1367] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.003s
I20240503 14:12:15.579793 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cb1bb0240e554fb2a420db2ec7d00e68" } new_table_name: "default.Table_2f39f2002afd4906a9f03f5dcce7e694" modify_external_catalogs: true
I20240503 14:12:15.580952 28600 raft_consensus.cc:348] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.581104 29516 leader_election.cc:288] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:15.581202 28600 raft_consensus.cc:374] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.581267 28600 raft_consensus.cc:729] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.581471 28600 consensus_queue.cc:260] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.582219 28600 ts_tablet_manager.cc:1398] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:15.582562 28600 tablet_bootstrap.cc:492] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.589138 28600 tablet_bootstrap.cc:654] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.592957 28661 log.cc:1196] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7c6f41bb9c5d40b0913264864ecc3528
I20240503 14:12:15.593432 28661 ts_tablet_manager.cc:1909] T 7c6f41bb9c5d40b0913264864ecc3528 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:15.593851 28614 tablet_bootstrap.cc:654] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.594333 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 520) took 1763 ms. Trace:
I20240503 14:12:15.594470 28661 rpcz_store.cc:276] 0503 14:12:13.830885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.832319 (+ 1434us) service_pool.cc:225] Handling call
0503 14:12:15.441087 (+1608768us) ts_tablet_manager.cc:1120] Deleting tablet 7c6f41bb9c5d40b0913264864ecc3528
0503 14:12:15.441096 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.450295 (+ 9199us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.477375 (+ 27080us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.594305 (+116930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608542}
I20240503 14:12:15.595021 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 517) took 1770 ms. Trace:
I20240503 14:12:15.595150 28661 rpcz_store.cc:276] 0503 14:12:13.825087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.826104 (+ 1017us) service_pool.cc:225] Handling call
0503 14:12:15.594863 (+1768759us) ts_tablet_manager.cc:1120] Deleting tablet ae1cd28bc07e4ccd9097f6dee471c5c6
0503 14:12:15.594872 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.595001 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741346}
I20240503 14:12:15.595549 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 515) took 1803 ms. Trace:
I20240503 14:12:15.595644 28661 rpcz_store.cc:276] 0503 14:12:13.792520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.796626 (+ 4106us) service_pool.cc:225] Handling call
0503 14:12:15.595431 (+1798805us) ts_tablet_manager.cc:1120] Deleting tablet f8fe462c9ee54a27b9714f6457036d76
0503 14:12:15.595440 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.595534 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1711961}
I20240503 14:12:15.595980 28661 tablet_replica.cc:330] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:15.596233 28661 raft_consensus.cc:2229] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:15.596597 28661 raft_consensus.cc:2258] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.596834 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dbb59d98834ab6bbac4abfa92681e3" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:15.597687 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dbb59d98834ab6bbac4abfa92681e3" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:15.597915 28214 raft_consensus.cc:3037] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.597955 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 451) took 3292 ms. Trace:
I20240503 14:12:15.598037 28653 rpcz_store.cc:276] 0503 14:12:12.305872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.332450 (+ 26578us) service_pool.cc:225] Handling call
0503 14:12:15.519973 (+3187523us) ts_tablet_manager.cc:1120] Deleting tablet 605531ed29214b1ca27011c5f5e96b11
0503 14:12:15.519983 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.541954 (+ 21971us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.550817 (+ 8863us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.597932 (+ 47115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187297}
I20240503 14:12:15.599228 28604 tablet_bootstrap.cc:492] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.599370 28604 ts_tablet_manager.cc:1367] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240503 14:12:15.600277 28604 raft_consensus.cc:348] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.600508 28604 raft_consensus.cc:374] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.600579 28604 raft_consensus.cc:729] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.600844 28604 consensus_queue.cc:260] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.601207 28653 tablet_replica.cc:330] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.601441 28653 raft_consensus.cc:2229] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.601625 28653 raft_consensus.cc:2258] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.602200 28653 ts_tablet_manager.cc:1875] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.604010 28604 ts_tablet_manager.cc:1398] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:15.604382 28604 tablet_bootstrap.cc:492] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.606446 28604 tablet_bootstrap.cc:654] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.607098 28603 ts_tablet_manager.cc:1888] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.607571 28603 log.cc:1196] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0b139ab0b5864024ac48215d46b0a7bd
I20240503 14:12:15.608804 28604 tablet_bootstrap.cc:492] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
W20240503 14:12:15.608739 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 50154.2us
I20240503 14:12:15.609020 28604 ts_tablet_manager.cc:1367] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
W20240503 14:12:15.609136 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d43823da09bb49be8f5d1a1149a4b11d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43823da09bb49be8f5d1a1149a4b11d already in progress: creating tablet
I20240503 14:12:15.609987 28604 raft_consensus.cc:348] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.609916 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:15.610673 28603 ts_tablet_manager.cc:1909] T 0b139ab0b5864024ac48215d46b0a7bd P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:15.610961 28214 raft_consensus.cc:2454] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:15.611522 28212 raft_consensus.cc:2454] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:15.617063 28653 ts_tablet_manager.cc:1888] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.617203 28653 log.cc:1196] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/962df44408484e3f8d7b7138f27c1a66
I20240503 14:12:15.617481 28653 ts_tablet_manager.cc:1909] T 962df44408484e3f8d7b7138f27c1a66 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.618412 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 453) took 3249 ms. Trace:
I20240503 14:12:15.618510 28653 rpcz_store.cc:276] 0503 14:12:12.369147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.369210 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:15.601143 (+3231933us) ts_tablet_manager.cc:1120] Deleting tablet 962df44408484e3f8d7b7138f27c1a66
0503 14:12:15.601157 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.613189 (+ 12032us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.616948 (+ 3759us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.618385 (+ 1437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3231671}
I20240503 14:12:15.618901 28653 tablet_replica.cc:330] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.618923 28614 tablet_bootstrap.cc:492] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.619051 28614 ts_tablet_manager.cc:1367] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20240503 14:12:15.619086 28653 raft_consensus.cc:2229] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.619238 28653 raft_consensus.cc:2258] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.619767 28653 ts_tablet_manager.cc:1875] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.612321 28661 ts_tablet_manager.cc:1875] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.620165 28604 raft_consensus.cc:374] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.620254 28604 raft_consensus.cc:729] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.620187 28614 raft_consensus.cc:348] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.620505 28614 raft_consensus.cc:374] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.620563 28614 raft_consensus.cc:729] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.620481 28604 consensus_queue.cc:260] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.620889 28614 consensus_queue.cc:260] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.621026 28604 ts_tablet_manager.cc:1398] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20240503 14:12:15.621480 28603 tablet_replica.cc:330] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:15.621644 28603 raft_consensus.cc:2229] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.621784 28603 raft_consensus.cc:2258] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.622520 28603 ts_tablet_manager.cc:1875] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.622664 28614 ts_tablet_manager.cc:1398] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:15.622749 28604 tablet_bootstrap.cc:492] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.623862 28614 tablet_bootstrap.cc:492] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.624625 28604 tablet_bootstrap.cc:654] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.627123 28604 tablet_bootstrap.cc:492] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.627256 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 1b008e3b655644d4a4f998df1fc7c1c9 (DEFAULT_TABLE table=default.Table_85c71baa763440bf939ea1193fa0af0c [id=9d717fbaf1ad46d7b6cc517324d7a636]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.627377 28604 ts_tablet_manager.cc:1367] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
W20240503 14:12:15.597659 28427 leader_election.cc:341] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.628810 28604 raft_consensus.cc:348] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.628969 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 1b008e3b655644d4a4f998df1fc7c1c9 (DEFAULT_TABLE table=default.Table_85c71baa763440bf939ea1193fa0af0c [id=9d717fbaf1ad46d7b6cc517324d7a636]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.629223 28604 raft_consensus.cc:374] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.629434 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b008e3b655644d4a4f998df1fc7c1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.629452 28604 raft_consensus.cc:729] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.629812 28604 consensus_queue.cc:260] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.630846 28604 ts_tablet_manager.cc:1398] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:15.631330 28604 tablet_bootstrap.cc:492] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.631803 28427 leader_election.cc:302] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.632253 29515 raft_consensus.cc:2781] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.632496 29515 raft_consensus.cc:686] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:15.632840 29515 consensus_queue.cc:237] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.633088 28661 ts_tablet_manager.cc:1888] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:15.633275 28661 log.cc:1196] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0f4fdf88fefa4ff7939f3ca05a82d7ba
I20240503 14:12:15.632402 28427 leader_election.cc:302] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.633780 29516 raft_consensus.cc:2781] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.633976 29516 raft_consensus.cc:686] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:15.634331 29516 consensus_queue.cc:237] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.637365 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 962df44408484e3f8d7b7138f27c1a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e4a544fc27624dcc8a4cb003e59408f2) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.640472 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 1b008e3b655644d4a4f998df1fc7c1c9 (DEFAULT_TABLE table=default.Table_85c71baa763440bf939ea1193fa0af0c [id=9d717fbaf1ad46d7b6cc517324d7a636]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.641112 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b008e3b655644d4a4f998df1fc7c1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.643083 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 66e36fcd0e894c34bec5b8d14449f12a (DEFAULT_TABLE table=default.Table_2f0f2612975448ecb7216f5e7b1dc2b0 [id=c37cdb9addf043939057219aef4a8cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.643957 28661 ts_tablet_manager.cc:1909] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:15.644986 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 523) took 1750 ms. Trace:
I20240503 14:12:15.646118 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e36fcd0e894c34bec5b8d14449f12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.645714 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 66e36fcd0e894c34bec5b8d14449f12a (DEFAULT_TABLE table=default.Table_2f0f2612975448ecb7216f5e7b1dc2b0 [id=c37cdb9addf043939057219aef4a8cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.646476 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e36fcd0e894c34bec5b8d14449f12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.650115 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b008e3b655644d4a4f998df1fc7c1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.653127 28603 ts_tablet_manager.cc:1888] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.657469 28604 tablet_bootstrap.cc:654] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.657860 28603 log.cc:1196] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0982b99a02ee48f2b6379e7be56c4c02
I20240503 14:12:15.658380 28603 ts_tablet_manager.cc:1909] T 0982b99a02ee48f2b6379e7be56c4c02 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:15.656241 28661 rpcz_store.cc:276] 0503 14:12:13.894724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.894778 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:15.595934 (+1701156us) ts_tablet_manager.cc:1120] Deleting tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba
0503 14:12:15.595943 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.623759 (+ 27816us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.632878 (+ 9119us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.644930 (+ 12052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1700926}
I20240503 14:12:15.660825 28600 tablet_bootstrap.cc:492] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.661039 28600 ts_tablet_manager.cc:1367] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.079s user 0.015s sys 0.018s
I20240503 14:12:15.661270 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 521) took 1766 ms. Trace:
I20240503 14:12:15.661410 28661 rpcz_store.cc:276] 0503 14:12:13.894410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.894499 (+ 89us) service_pool.cc:225] Handling call
0503 14:12:15.661080 (+1766581us) ts_tablet_manager.cc:1120] Deleting tablet 920b41e3b21347b393497de4d4e87469
0503 14:12:15.661091 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.661229 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766015}
I20240503 14:12:15.661813 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 525) took 1766 ms. Trace:
I20240503 14:12:15.662119 28600 raft_consensus.cc:348] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.662285 28661 rpcz_store.cc:276] 0503 14:12:13.895030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.895085 (+ 55us) service_pool.cc:225] Handling call
0503 14:12:15.661663 (+1766578us) ts_tablet_manager.cc:1120] Deleting tablet 9b21cd1262d94d1c994b29c44f86cb5e
0503 14:12:15.661672 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.661781 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766326}
I20240503 14:12:15.662521 28600 raft_consensus.cc:374] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.662592 28661 tablet_replica.cc:330] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:15.662685 28600 raft_consensus.cc:729] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.662818 28661 raft_consensus.cc:2229] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:15.663040 28600 consensus_queue.cc:260] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.663240 28661 raft_consensus.cc:2258] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.669358 28614 tablet_bootstrap.cc:654] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.670787 28661 ts_tablet_manager.cc:1875] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.674371 28653 ts_tablet_manager.cc:1888] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.674546 28653 log.cc:1196] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/37dc7d96357743089372baf71654b1e3
I20240503 14:12:15.677120 28653 ts_tablet_manager.cc:1909] T 37dc7d96357743089372baf71654b1e3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.678580 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 454) took 3300 ms. Trace:
I20240503 14:12:15.679775 28653 rpcz_store.cc:276] 0503 14:12:12.378420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.378881 (+ 461us) service_pool.cc:225] Handling call
0503 14:12:15.618848 (+3239967us) ts_tablet_manager.cc:1120] Deleting tablet 37dc7d96357743089372baf71654b1e3
0503 14:12:15.618858 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.669603 (+ 50745us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.674234 (+ 4631us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.678548 (+ 4314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3237347}
I20240503 14:12:15.680824 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 455) took 3270 ms. Trace:
I20240503 14:12:15.681341 28653 rpcz_store.cc:276] 0503 14:12:12.410158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.410222 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:15.680642 (+3270420us) ts_tablet_manager.cc:1120] Deleting tablet daf5d153ab5142c0902cc6e96629f412
0503 14:12:15.680656 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.680800 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3270124}
I20240503 14:12:15.682686 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 456) took 3259 ms. Trace:
I20240503 14:12:15.683024 28653 rpcz_store.cc:276] 0503 14:12:12.422952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.445738 (+ 22786us) service_pool.cc:225] Handling call
0503 14:12:15.682098 (+3236360us) ts_tablet_manager.cc:1120] Deleting tablet 3d137ef4cfc2476cbc70f1cc72ad89e6
0503 14:12:15.682112 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.682663 (+ 551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3236034}
I20240503 14:12:15.683502 28600 ts_tablet_manager.cc:1398] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 14:12:15.684154 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 457) took 3227 ms. Trace:
I20240503 14:12:15.684767 28653 rpcz_store.cc:276] 0503 14:12:12.456479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.459258 (+ 2779us) service_pool.cc:225] Handling call
0503 14:12:15.683904 (+3224646us) ts_tablet_manager.cc:1120] Deleting tablet c816f3b05cfb4496984f6cb28d310cc4
0503 14:12:15.683915 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.684131 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3224135}
I20240503 14:12:15.685570 28653 tablet_replica.cc:330] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.686026 28653 raft_consensus.cc:2229] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.686636 28653 raft_consensus.cc:2258] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.687435 28653 ts_tablet_manager.cc:1875] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.689980 28600 tablet_bootstrap.cc:492] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.691464 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 66e36fcd0e894c34bec5b8d14449f12a (DEFAULT_TABLE table=default.Table_2f0f2612975448ecb7216f5e7b1dc2b0 [id=c37cdb9addf043939057219aef4a8cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.692232 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e36fcd0e894c34bec5b8d14449f12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.692610 28600 tablet_bootstrap.cc:654] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.693188 29281 raft_consensus.cc:482] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.693322 29281 raft_consensus.cc:504] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.694572 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c924a74ee836406fb3157baef6f4c901" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:15.694754 28614 tablet_bootstrap.cc:492] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.694909 28614 ts_tablet_manager.cc:1367] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.000s
I20240503 14:12:15.695420 29281 leader_election.cc:288] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:15.695186 28150 leader_election.cc:341] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.694887 29573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.253:44259 every 8 connection(s)
I20240503 14:12:15.695843 28614 raft_consensus.cc:348] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.696053 28614 raft_consensus.cc:374] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.696200 28614 raft_consensus.cc:729] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.696264 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c924a74ee836406fb3157baef6f4c901" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:15.696627 28614 consensus_queue.cc:260] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:15.696787 28150 leader_election.cc:341] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:15.696897 28150 leader_election.cc:302] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:15.697286 29281 raft_consensus.cc:2726] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.697455 29295 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.253:44259
I20240503 14:12:15.701205 29295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-1/data/info.pb
I20240503 14:12:15.701634 28653 ts_tablet_manager.cc:1888] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.701795 28653 log.cc:1196] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a6dbb59d98834ab6bbac4abfa92681e3
I20240503 14:12:15.702101 28653 ts_tablet_manager.cc:1909] T a6dbb59d98834ab6bbac4abfa92681e3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.703543 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 29295
I20240503 14:12:15.707976 28614 ts_tablet_manager.cc:1398] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:12:15.708912 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 94c95ac1e0934710b6d3b5589ceb9346 (DEFAULT_TABLE table=default.Table_fb7b6285a62e4ccaad9e7f7a3d82c1aa [id=f8f91946198a406582d2df57b6964165]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.709350 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c95ac1e0934710b6d3b5589ceb9346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.710726 28614 tablet_bootstrap.cc:492] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.712500 28614 tablet_bootstrap.cc:654] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.718175 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 3112560e16f14b4e89aae90c014be9fb (DEFAULT_TABLE table=default.Table_6de7dc45944845628d02aa20bfa0dcad [id=752542de6afc4e5ebeb198311c314e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.717773 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 3112560e16f14b4e89aae90c014be9fb (DEFAULT_TABLE table=default.Table_6de7dc45944845628d02aa20bfa0dcad [id=752542de6afc4e5ebeb198311c314e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.718670 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3112560e16f14b4e89aae90c014be9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.718864 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3112560e16f14b4e89aae90c014be9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.719355 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 2946ba624b244db9a617f2e202bfc1ec (DEFAULT_TABLE table=default.Table_cfcbfe39af47450d9a0917259530eb81 [id=29b2ba9909c34f528bdbb8ef06268373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.719712 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2946ba624b244db9a617f2e202bfc1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.719811 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 94c95ac1e0934710b6d3b5589ceb9346 (DEFAULT_TABLE table=default.Table_fb7b6285a62e4ccaad9e7f7a3d82c1aa [id=f8f91946198a406582d2df57b6964165]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.720289 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c95ac1e0934710b6d3b5589ceb9346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.726253 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 2946ba624b244db9a617f2e202bfc1ec (DEFAULT_TABLE table=default.Table_cfcbfe39af47450d9a0917259530eb81 [id=29b2ba9909c34f528bdbb8ef06268373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.726778 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2946ba624b244db9a617f2e202bfc1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.718426 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 3112560e16f14b4e89aae90c014be9fb (DEFAULT_TABLE table=default.Table_6de7dc45944845628d02aa20bfa0dcad [id=752542de6afc4e5ebeb198311c314e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.729038 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3112560e16f14b4e89aae90c014be9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.720391 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 2946ba624b244db9a617f2e202bfc1ec (DEFAULT_TABLE table=default.Table_cfcbfe39af47450d9a0917259530eb81 [id=29b2ba9909c34f528bdbb8ef06268373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.733528 28614 tablet_bootstrap.cc:492] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.733711 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2946ba624b244db9a617f2e202bfc1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.734254 28614 ts_tablet_manager.cc:1367] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
W20240503 14:12:15.736413 28150 consensus_peers.cc:485] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c6f41bb9c5d40b0913264864ecc3528. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:15.735124 28614 raft_consensus.cc:348] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.737227 28614 raft_consensus.cc:374] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.737293 28614 raft_consensus.cc:729] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.738263 28614 consensus_queue.cc:260] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.741147 28614 ts_tablet_manager.cc:1398] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:15.741602 28614 tablet_bootstrap.cc:492] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.746125 28600 tablet_bootstrap.cc:492] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.748288 28600 ts_tablet_manager.cc:1367] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20240503 14:12:15.749270 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:15.751487 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 458) took 3288 ms. Trace:
I20240503 14:12:15.751878 28653 rpcz_store.cc:276] 0503 14:12:12.461691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.463295 (+ 1604us) service_pool.cc:225] Handling call
0503 14:12:15.685516 (+3222221us) ts_tablet_manager.cc:1120] Deleting tablet a6dbb59d98834ab6bbac4abfa92681e3
0503 14:12:15.685527 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.697679 (+ 12152us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.701506 (+ 3827us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.749754 (+ 48248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3221952}
I20240503 14:12:15.753054 28600 raft_consensus.cc:348] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.757903 29575 sys_catalog.cc:263] Verifying existing consensus state
I20240503 14:12:15.758411 28614 tablet_bootstrap.cc:654] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.753082 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 467) took 3208 ms. Trace:
I20240503 14:12:15.758690 28653 rpcz_store.cc:276] 0503 14:12:12.544719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.544782 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:15.752902 (+3208120us) ts_tablet_manager.cc:1120] Deleting tablet 007a892ef8504b848a400c05740eaa0c
0503 14:12:15.752914 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.753062 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207861}
I20240503 14:12:15.758601 28600 raft_consensus.cc:374] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.759397 28600 raft_consensus.cc:729] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.759721 28600 consensus_queue.cc:260] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.762324 28600 ts_tablet_manager.cc:1398] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240503 14:12:15.762686 28600 tablet_bootstrap.cc:492] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.764557 28600 tablet_bootstrap.cc:654] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.764511 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 605531ed29214b1ca27011c5f5e96b11 (table default.Table_2907862ac0af4c3692460a0071e8bce3 [id=651246fc28e144af9b8a0bbe9926aa22]) successfully deleted
I20240503 14:12:15.764851 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 962df44408484e3f8d7b7138f27c1a66 (table default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]) successfully deleted
I20240503 14:12:15.765205 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 37dc7d96357743089372baf71654b1e3 (table default.Table_85ee5bde7ad044389efb107eefd39fd3 [id=11af424ea24549bc83f6203448ae619c]) successfully deleted
W20240503 14:12:15.765583 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet daf5d153ab5142c0902cc6e96629f412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf5d153ab5142c0902cc6e96629f412 already in progress: creating tablet
W20240503 14:12:15.765838 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 already in progress: creating tablet
I20240503 14:12:15.765921 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 471) took 3163 ms. Trace:
W20240503 14:12:15.766099 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c816f3b05cfb4496984f6cb28d310cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c816f3b05cfb4496984f6cb28d310cc4 already in progress: creating tablet
I20240503 14:12:15.766037 28653 rpcz_store.cc:276] 0503 14:12:12.602442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.602607 (+ 165us) service_pool.cc:225] Handling call
0503 14:12:15.765737 (+3163130us) ts_tablet_manager.cc:1120] Deleting tablet d4c8a97076a44f9eaa3a15026f90a304
0503 14:12:15.765748 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.765891 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3117206}
I20240503 14:12:15.766674 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 470) took 3164 ms. Trace:
I20240503 14:12:15.766829 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a6dbb59d98834ab6bbac4abfa92681e3 (table default.Table_33f62889c4784d6cb42ac76faa20113c [id=c62152292eb14f4aaae391155d12da6f]) successfully deleted
I20240503 14:12:15.766773 28653 rpcz_store.cc:276] 0503 14:12:12.602222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.602507 (+ 285us) service_pool.cc:225] Handling call
0503 14:12:15.766535 (+3164028us) ts_tablet_manager.cc:1120] Deleting tablet b476e87faea1415d95ae014a93fa22af
0503 14:12:15.766545 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.766656 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3117400}
I20240503 14:12:15.767204 28653 tablet_replica.cc:330] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:15.767239 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d4c8a97076a44f9eaa3a15026f90a304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c8a97076a44f9eaa3a15026f90a304 already in progress: creating tablet
W20240503 14:12:15.767508 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b476e87faea1415d95ae014a93fa22af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b476e87faea1415d95ae014a93fa22af already in progress: creating tablet
I20240503 14:12:15.767544 28653 raft_consensus.cc:2229] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.767827 28653 raft_consensus.cc:2258] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.768556 28653 ts_tablet_manager.cc:1875] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.768904 28604 tablet_bootstrap.cc:492] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.769145 28604 ts_tablet_manager.cc:1367] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.138s user 0.010s sys 0.013s
I20240503 14:12:15.770047 28604 raft_consensus.cc:348] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.770344 28604 raft_consensus.cc:374] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.770455 28604 raft_consensus.cc:729] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.770793 28604 consensus_queue.cc:260] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.771481 28604 ts_tablet_manager.cc:1398] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:15.771735 28614 tablet_bootstrap.cc:492] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.774045 28614 ts_tablet_manager.cc:1367] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 14:12:15.774436 28604 tablet_bootstrap.cc:492] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.775642 28653 ts_tablet_manager.cc:1888] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.775573 28614 raft_consensus.cc:348] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.776386 28604 tablet_bootstrap.cc:654] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.778710 28614 raft_consensus.cc:374] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.784493 28653 log.cc:1196] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/1044f34588d045ce898435b61a90503f
I20240503 14:12:15.785234 28653 ts_tablet_manager.cc:1909] T 1044f34588d045ce898435b61a90503f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.786325 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:15.789237 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 472) took 3070 ms. Trace:
I20240503 14:12:15.789417 28614 raft_consensus.cc:729] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.790158 28614 consensus_queue.cc:260] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.790604 28653 rpcz_store.cc:276] 0503 14:12:12.718522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.718927 (+ 405us) service_pool.cc:225] Handling call
0503 14:12:15.767149 (+3048222us) ts_tablet_manager.cc:1120] Deleting tablet 1044f34588d045ce898435b61a90503f
0503 14:12:15.767159 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.772242 (+ 5083us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.775507 (+ 3265us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.789201 (+ 13694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3047933}
I20240503 14:12:15.791839 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 473) took 3069 ms. Trace:
I20240503 14:12:15.792512 28653 rpcz_store.cc:276] 0503 14:12:12.722401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.722470 (+ 69us) service_pool.cc:225] Handling call
0503 14:12:15.791656 (+3069186us) ts_tablet_manager.cc:1120] Deleting tablet 7b6c998e518541248a87e4c1551ec080
0503 14:12:15.791669 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.791815 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3067986}
I20240503 14:12:15.793478 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 474) took 3069 ms. Trace:
I20240503 14:12:15.793967 28653 rpcz_store.cc:276] 0503 14:12:12.723862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:12.727440 (+ 3578us) service_pool.cc:225] Handling call
0503 14:12:15.793339 (+3065899us) ts_tablet_manager.cc:1120] Deleting tablet ce4846d662a64d51b16fd5447c371e9a
0503 14:12:15.793351 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.793460 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3025122}
I20240503 14:12:15.794963 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 491) took 2506 ms. Trace:
I20240503 14:12:15.795444 28653 rpcz_store.cc:276] 0503 14:12:13.288942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.289903 (+ 961us) service_pool.cc:225] Handling call
0503 14:12:15.794816 (+2504913us) ts_tablet_manager.cc:1120] Deleting tablet 4084bd4fbf604042bb33ff03c3f8cc19
0503 14:12:15.794829 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.794944 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2504587}
I20240503 14:12:15.796233 28653 tablet_replica.cc:330] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.796696 28653 raft_consensus.cc:2229] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.797206 28653 raft_consensus.cc:2258] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.797592 28614 ts_tablet_manager.cc:1398] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240503 14:12:15.798126 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:15.798324 28653 ts_tablet_manager.cc:1875] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.804759 28614 tablet_bootstrap.cc:492] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.808521 29515 raft_consensus.cc:482] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.808730 29515 raft_consensus.cc:504] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.809525 29515 leader_election.cc:288] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:15.810457 28427 leader_election.cc:341] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 962df44408484e3f8d7b7138f27c1a66
W20240503 14:12:15.810678 28427 leader_election.cc:341] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 962df44408484e3f8d7b7138f27c1a66
I20240503 14:12:15.810763 28427 leader_election.cc:302] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.811002 29515 raft_consensus.cc:2726] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:15.814682 28614 tablet_bootstrap.cc:654] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.817130 28600 tablet_bootstrap.cc:492] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.817399 28600 ts_tablet_manager.cc:1367] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.000s
I20240503 14:12:15.818230 28600 raft_consensus.cc:348] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.818583 28600 raft_consensus.cc:374] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.818684 28600 raft_consensus.cc:729] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.819013 28600 consensus_queue.cc:260] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.821884 28600 ts_tablet_manager.cc:1398] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:15.822279 28600 tablet_bootstrap.cc:492] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.823305 29515 raft_consensus.cc:482] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.823459 29515 raft_consensus.cc:504] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.823599 28264 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:15.824201 29515 leader_election.cc:288] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:15.824820 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Bootstrap starting.
I20240503 14:12:15.826103 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7c6f41bb9c5d40b0913264864ecc3528 (table default.Table_6add22bd3ca9452fa98a991f4e9b817f [id=806d4f0bc4524d9481487418ff797a5d]) successfully deleted
W20240503 14:12:15.827288 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f8fe462c9ee54a27b9714f6457036d76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8fe462c9ee54a27b9714f6457036d76 already in progress: creating tablet
I20240503 14:12:15.827652 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dc7d96357743089372baf71654b1e3" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:15.828380 28427 leader_election.cc:341] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:15.828613 28427 leader_election.cc:341] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 37dc7d96357743089372baf71654b1e3
I20240503 14:12:15.828711 28427 leader_election.cc:302] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:15.829005 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba (table default.Table_a888de5f6a40467db886c3141d2a87cc [id=caca4dbaa701453794ac56475f46704e]) successfully deleted
I20240503 14:12:15.829035 29515 raft_consensus.cc:2726] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:15.829315 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 920b41e3b21347b393497de4d4e87469 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920b41e3b21347b393497de4d4e87469 already in progress: creating tablet
I20240503 14:12:15.829558 28600 tablet_bootstrap.cc:654] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:15.829598 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9b21cd1262d94d1c994b29c44f86cb5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b21cd1262d94d1c994b29c44f86cb5e already in progress: creating tablet
I20240503 14:12:15.831539 28325 tablet_service.cc:1467] Processing CreateTablet for tablet af0137d7b1034a79bf4947c7d3f3fd57 (DEFAULT_TABLE table=default.Table_a9e8bd96c6824e9c8c64036f0848c876 [id=9e438ac1c3254c7aae6f81c707f433d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.832108 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0137d7b1034a79bf4947c7d3f3fd57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.830679 28179 tablet_service.cc:1467] Processing CreateTablet for tablet af0137d7b1034a79bf4947c7d3f3fd57 (DEFAULT_TABLE table=default.Table_a9e8bd96c6824e9c8c64036f0848c876 [id=9e438ac1c3254c7aae6f81c707f433d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.832530 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0137d7b1034a79bf4947c7d3f3fd57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.832592 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:15.842661 28661 ts_tablet_manager.cc:1888] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:15.842970 28661 log.cc:1196] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7945118bb17b431dac8670007d2d0a18
I20240503 14:12:15.843621 28661 ts_tablet_manager.cc:1909] T 7945118bb17b431dac8670007d2d0a18 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:15.844714 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 518) took 2019 ms. Trace:
I20240503 14:12:15.844929 28661 rpcz_store.cc:276] 0503 14:12:13.825295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.825603 (+ 308us) service_pool.cc:225] Handling call
0503 14:12:15.662528 (+1836925us) ts_tablet_manager.cc:1120] Deleting tablet 7945118bb17b431dac8670007d2d0a18
0503 14:12:15.662537 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.757362 (+ 94825us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.821061 (+ 63699us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.844683 (+ 23622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766320}
I20240503 14:12:15.845764 28661 tablet_replica.cc:330] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:15.846058 28661 raft_consensus.cc:2229] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:15.846508 28661 pending_rounds.cc:62] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:15.847069 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 94c95ac1e0934710b6d3b5589ceb9346 (DEFAULT_TABLE table=default.Table_fb7b6285a62e4ccaad9e7f7a3d82c1aa [id=f8f91946198a406582d2df57b6964165]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.847744 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c95ac1e0934710b6d3b5589ceb9346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.848141 28653 ts_tablet_manager.cc:1888] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.850306 28614 tablet_bootstrap.cc:492] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.850601 28614 ts_tablet_manager.cc:1367] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.003s
I20240503 14:12:15.851814 28653 log.cc:1196] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ca6627f0d9a544bb8176bea6c1af5ab4
I20240503 14:12:15.851619 28614 raft_consensus.cc:348] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.851910 28614 raft_consensus.cc:374] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.851974 28614 raft_consensus.cc:729] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.852285 28614 consensus_queue.cc:260] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.852921 28653 ts_tablet_manager.cc:1909] T ca6627f0d9a544bb8176bea6c1af5ab4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.854704 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:15.855124 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:15.856585 28614 ts_tablet_manager.cc:1398] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:15.853842 28600 tablet_bootstrap.cc:492] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.859592 28600 ts_tablet_manager.cc:1367] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.003s
I20240503 14:12:15.860556 28614 tablet_bootstrap.cc:492] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.860599 28600 raft_consensus.cc:348] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.860810 28600 raft_consensus.cc:374] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.860862 28600 raft_consensus.cc:729] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.861126 28600 consensus_queue.cc:260] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.863430 28600 ts_tablet_manager.cc:1398] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:15.859233 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 492) took 2570 ms. Trace:
I20240503 14:12:15.863698 28653 rpcz_store.cc:276] 0503 14:12:13.289103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.290268 (+ 1165us) service_pool.cc:225] Handling call
0503 14:12:15.796181 (+2505913us) ts_tablet_manager.cc:1120] Deleting tablet ca6627f0d9a544bb8176bea6c1af5ab4
0503 14:12:15.796191 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.831556 (+ 35365us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.847989 (+ 16433us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.859200 (+ 11211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2505643}
I20240503 14:12:15.864470 28614 tablet_bootstrap.cc:654] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.864317 28600 tablet_bootstrap.cc:492] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.866139 28600 tablet_bootstrap.cc:654] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.866508 28661 pending_rounds.cc:68] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597706927095808 op_type: NO_OP noop_request { }
I20240503 14:12:15.866699 28661 raft_consensus.cc:2866] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:15.866811 28661 raft_consensus.cc:2258] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.866425 28653 tablet_replica.cc:330] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.867451 28661 ts_tablet_manager.cc:1875] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.867794 28653 raft_consensus.cc:2229] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.868129 28653 raft_consensus.cc:2258] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.868777 28653 ts_tablet_manager.cc:1875] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.879328 28604 tablet_bootstrap.cc:492] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.879608 28604 ts_tablet_manager.cc:1367] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.105s user 0.037s sys 0.020s
I20240503 14:12:15.880774 28469 tablet_service.cc:1467] Processing CreateTablet for tablet af0137d7b1034a79bf4947c7d3f3fd57 (DEFAULT_TABLE table=default.Table_a9e8bd96c6824e9c8c64036f0848c876 [id=9e438ac1c3254c7aae6f81c707f433d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.881210 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0137d7b1034a79bf4947c7d3f3fd57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.881384 28604 raft_consensus.cc:348] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.881567 28604 raft_consensus.cc:374] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.881616 28604 raft_consensus.cc:729] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.881872 28604 consensus_queue.cc:260] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.882625 28600 tablet_bootstrap.cc:492] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:15.882894 28600 ts_tablet_manager.cc:1367] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240503 14:12:15.883898 28600 raft_consensus.cc:348] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.884074 28604 ts_tablet_manager.cc:1398] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:15.884142 28600 raft_consensus.cc:374] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.884202 28600 raft_consensus.cc:729] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.884455 28604 tablet_bootstrap.cc:492] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.884472 28600 consensus_queue.cc:260] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.887063 28600 ts_tablet_manager.cc:1398] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:15.892114 28604 tablet_bootstrap.cc:654] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.895162 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc7bb4b52984487aa15a8a7e98ae3ca (DEFAULT_TABLE table=default.Table_0b5dae96b3b64066ba7334dc73e578e8 [id=b0703ec084d74051a574021c92cf47c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.895349 28614 tablet_bootstrap.cc:492] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.896258 28614 ts_tablet_manager.cc:1367] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.002s
I20240503 14:12:15.896627 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc7bb4b52984487aa15a8a7e98ae3ca (DEFAULT_TABLE table=default.Table_0b5dae96b3b64066ba7334dc73e578e8 [id=b0703ec084d74051a574021c92cf47c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.895679 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc7bb4b52984487aa15a8a7e98ae3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.897112 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc7bb4b52984487aa15a8a7e98ae3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.900130 28600 tablet_bootstrap.cc:492] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:15.900861 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc7bb4b52984487aa15a8a7e98ae3ca (DEFAULT_TABLE table=default.Table_0b5dae96b3b64066ba7334dc73e578e8 [id=b0703ec084d74051a574021c92cf47c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.900839 28614 raft_consensus.cc:348] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:15.901069 28614 raft_consensus.cc:374] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.901129 28614 raft_consensus.cc:729] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.901288 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc7bb4b52984487aa15a8a7e98ae3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.901355 28614 consensus_queue.cc:260] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:15.901834 28600 tablet_bootstrap.cc:654] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.905503 28614 ts_tablet_manager.cc:1398] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:12:15.907089 28653 ts_tablet_manager.cc:1888] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:15.907235 28653 log.cc:1196] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/651478ef632144079f93f3b50bff4611
I20240503 14:12:15.907636 28653 ts_tablet_manager.cc:1909] T 651478ef632144079f93f3b50bff4611 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.908008 28614 tablet_bootstrap.cc:492] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.908587 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 494) took 2619 ms. Trace:
I20240503 14:12:15.912811 28333 tablet_service.cc:1467] Processing CreateTablet for tablet f5bea5960cbb44d58d1dbabc888903c2 (DEFAULT_TABLE table=default.Table_f40d0e0c2ffb4c3a8169abdfdc3f8148 [id=5f179836a86f45b4b01c2ab108aac8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.913137 28604 tablet_bootstrap.cc:492] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.913295 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bea5960cbb44d58d1dbabc888903c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.913359 28604 ts_tablet_manager.cc:1367] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20240503 14:12:15.913151 28653 rpcz_store.cc:276] 0503 14:12:13.289429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.290663 (+ 1234us) service_pool.cc:225] Handling call
0503 14:12:15.866369 (+2575706us) ts_tablet_manager.cc:1120] Deleting tablet 651478ef632144079f93f3b50bff4611
0503 14:12:15.866380 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.877434 (+ 11054us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.906942 (+ 29508us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.908560 (+ 1618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2575502}
I20240503 14:12:15.913959 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 490) took 2626 ms. Trace:
I20240503 14:12:15.914081 28653 rpcz_store.cc:276] 0503 14:12:13.287129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.287228 (+ 99us) service_pool.cc:225] Handling call
0503 14:12:15.913791 (+2626563us) ts_tablet_manager.cc:1120] Deleting tablet 320c526e845e4dc78da345bcb095918f
0503 14:12:15.913804 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.913936 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620254}
I20240503 14:12:15.914237 28604 raft_consensus.cc:348] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.914440 28604 raft_consensus.cc:374] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.914485 28604 raft_consensus.cc:729] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.914592 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 493) took 2625 ms. Trace:
I20240503 14:12:15.914700 28604 consensus_queue.cc:260] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.914682 28653 rpcz_store.cc:276] 0503 14:12:13.289291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.290575 (+ 1284us) service_pool.cc:225] Handling call
0503 14:12:15.914467 (+2623892us) ts_tablet_manager.cc:1120] Deleting tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df
0503 14:12:15.914477 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.914573 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620737}
I20240503 14:12:15.915692 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 495) took 2421 ms. Trace:
I20240503 14:12:15.915778 28653 rpcz_store.cc:276] 0503 14:12:13.493877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.521267 (+ 27390us) service_pool.cc:225] Handling call
0503 14:12:15.915545 (+2394278us) ts_tablet_manager.cc:1120] Deleting tablet a6c17cfce7e644668817bf3483a098d4
0503 14:12:15.915554 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.915660 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393916}
I20240503 14:12:15.916941 28653 tablet_replica.cc:330] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.916965 28604 ts_tablet_manager.cc:1398] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:15.917423 28653 raft_consensus.cc:2229] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:15.917857 28653 raft_consensus.cc:2258] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.918699 28653 ts_tablet_manager.cc:1875] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.919118 28604 tablet_bootstrap.cc:492] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:15.925413 28604 tablet_bootstrap.cc:654] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.927031 28458 tablet_service.cc:1467] Processing CreateTablet for tablet f5bea5960cbb44d58d1dbabc888903c2 (DEFAULT_TABLE table=default.Table_f40d0e0c2ffb4c3a8169abdfdc3f8148 [id=5f179836a86f45b4b01c2ab108aac8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.927549 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bea5960cbb44d58d1dbabc888903c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.922457 28614 tablet_bootstrap.cc:654] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.927726 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8bf34e96e04868b135a1a9422a46be (DEFAULT_TABLE table=default.Table_4e13e0a790bc495f93e8cabc45270a80 [id=640fb8f7e45e42a1a5030e730e150ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.928109 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8bf34e96e04868b135a1a9422a46be (DEFAULT_TABLE table=default.Table_4e13e0a790bc495f93e8cabc45270a80 [id=640fb8f7e45e42a1a5030e730e150ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.928227 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8bf34e96e04868b135a1a9422a46be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.928500 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8bf34e96e04868b135a1a9422a46be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.927327 28179 tablet_service.cc:1467] Processing CreateTablet for tablet f5bea5960cbb44d58d1dbabc888903c2 (DEFAULT_TABLE table=default.Table_f40d0e0c2ffb4c3a8169abdfdc3f8148 [id=5f179836a86f45b4b01c2ab108aac8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.935469 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bea5960cbb44d58d1dbabc888903c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.936571 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8bf34e96e04868b135a1a9422a46be (DEFAULT_TABLE table=default.Table_4e13e0a790bc495f93e8cabc45270a80 [id=640fb8f7e45e42a1a5030e730e150ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.937247 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8bf34e96e04868b135a1a9422a46be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.938501 28614 tablet_bootstrap.cc:492] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:15.938767 28614 ts_tablet_manager.cc:1367] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20240503 14:12:15.939880 28614 raft_consensus.cc:348] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.940292 28614 raft_consensus.cc:374] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.940410 28614 raft_consensus.cc:729] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.940768 28614 consensus_queue.cc:260] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.945829 28614 ts_tablet_manager.cc:1398] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:12:15.945977 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 59b7628bcae140939184428ed15dda93 (DEFAULT_TABLE table=default.Table_29aaa3a8d8ed47dab6dfc37b3989f427 [id=9fdc238ef0914248814d154970c52b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.946748 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b7628bcae140939184428ed15dda93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.947687 28614 tablet_bootstrap.cc:492] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:15.950505 28614 tablet_bootstrap.cc:654] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.955597 28653 ts_tablet_manager.cc:1888] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:15.960778 28604 tablet_bootstrap.cc:492] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:15.961117 28604 ts_tablet_manager.cc:1367] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.015s
I20240503 14:12:15.961803 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 59b7628bcae140939184428ed15dda93 (DEFAULT_TABLE table=default.Table_29aaa3a8d8ed47dab6dfc37b3989f427 [id=9fdc238ef0914248814d154970c52b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.962409 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b7628bcae140939184428ed15dda93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.962358 28604 raft_consensus.cc:348] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.962636 28327 tablet_service.cc:1467] Processing CreateTablet for tablet d6720048b4eb4201a75c15cf8031069f (DEFAULT_TABLE table=default.Table_a635bdd722254d13bf665ab69c76c554 [id=bf4ff0c5bfa44facaf0cc49a477d1a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.963086 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6720048b4eb4201a75c15cf8031069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.963115 28458 tablet_service.cc:1467] Processing CreateTablet for tablet d6720048b4eb4201a75c15cf8031069f (DEFAULT_TABLE table=default.Table_a635bdd722254d13bf665ab69c76c554 [id=bf4ff0c5bfa44facaf0cc49a477d1a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.963572 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6720048b4eb4201a75c15cf8031069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.963474 28194 tablet_service.cc:1467] Processing CreateTablet for tablet d6720048b4eb4201a75c15cf8031069f (DEFAULT_TABLE table=default.Table_a635bdd722254d13bf665ab69c76c554 [id=bf4ff0c5bfa44facaf0cc49a477d1a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.963999 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6720048b4eb4201a75c15cf8031069f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:15.967716 28427 consensus_peers.cc:485] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1044f34588d045ce898435b61a90503f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:15.970619 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 59b7628bcae140939184428ed15dda93 (DEFAULT_TABLE table=default.Table_29aaa3a8d8ed47dab6dfc37b3989f427 [id=9fdc238ef0914248814d154970c52b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.962597 28604 raft_consensus.cc:374] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.971145 28604 raft_consensus.cc:729] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.971338 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b7628bcae140939184428ed15dda93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.971479 28604 consensus_queue.cc:260] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.976198 28604 ts_tablet_manager.cc:1398] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 14:12:15.976235 28653 log.cc:1196] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/50edc38cdbe74397aa4f4928d56d9fbc
I20240503 14:12:15.976338 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 393f27164887427d8a654c51cfc04e14 (DEFAULT_TABLE table=default.Table_b65e95f8090a40e5abb366c2aa904d60 [id=3ffa9f842d2b49b08e1c08fe6878eaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.976609 28653 ts_tablet_manager.cc:1909] T 50edc38cdbe74397aa4f4928d56d9fbc P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:15.976948 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393f27164887427d8a654c51cfc04e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.977490 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 499) took 2436 ms. Trace:
I20240503 14:12:15.977625 28653 rpcz_store.cc:276] 0503 14:12:13.541087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.544625 (+ 3538us) service_pool.cc:225] Handling call
0503 14:12:15.916889 (+2372264us) ts_tablet_manager.cc:1120] Deleting tablet 50edc38cdbe74397aa4f4928d56d9fbc
0503 14:12:15.916899 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.946447 (+ 29548us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.955450 (+ 9003us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:15.977461 (+ 22011us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2371961}
I20240503 14:12:15.979606 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 50edc38cdbe74397aa4f4928d56d9fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:15.980044 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0b139ab0b5864024ac48215d46b0a7bd (table default.Table_52932a5c26744566bfc744e2ed1d8aee [id=5c194c0fc1c94da0a40f0f3f0d7484b7]) successfully deleted
I20240503 14:12:15.980811 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 393f27164887427d8a654c51cfc04e14 (DEFAULT_TABLE table=default.Table_b65e95f8090a40e5abb366c2aa904d60 [id=3ffa9f842d2b49b08e1c08fe6878eaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.981326 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393f27164887427d8a654c51cfc04e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.983563 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 504) took 2394 ms. Trace:
I20240503 14:12:15.983672 28653 rpcz_store.cc:276] 0503 14:12:13.589556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.589787 (+ 231us) service_pool.cc:225] Handling call
0503 14:12:15.983402 (+2393615us) ts_tablet_manager.cc:1120] Deleting tablet 34fa067359a640eeba8c80d08371904e
0503 14:12:15.983415 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.983536 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393352}
W20240503 14:12:15.984287 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df63f75c157a4e928d4dbbda06cfa5ce consensus state has no leader
I20240503 14:12:15.985126 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 31ddd791d4e14630bfb3b4b1513f76e1
I20240503 14:12:15.986060 28653 tablet_replica.cc:330] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:15.986085 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet df63f75c157a4e928d4dbbda06cfa5ce (table default.Table_13b6a3c3a46b4308b4ce309165333325 [id=d29fe68058a14e178e42a10043a91738]). Expected version 2 got 0
I20240503 14:12:15.986375 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:15.986337 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c8a07599c3614ce2ba13285c679960bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:15.987056 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet a1ebec4622ef4950bd1c70451958c59b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:15.987123 28653 raft_consensus.cc:2229] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:15.987588 28653 pending_rounds.cc:62] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:15.987648 28653 pending_rounds.cc:68] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597707224031232 op_type: NO_OP noop_request { }
I20240503 14:12:15.987777 28653 raft_consensus.cc:2866] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:15.987859 28653 raft_consensus.cc:2258] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:15.987742 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c64ecffdd4a84ba5bc64aba80e1de1ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.988412 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f75e102dd4249c3800e1fcb4339acf3 consensus state has no leader
I20240503 14:12:15.988548 28653 ts_tablet_manager.cc:1875] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:15.988864 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_713837a8f6e7403fa453e5203654186b" } new_table_name: "default.Table_16583130ff3c4792a96fb0039f73843f" modify_external_catalogs: true
I20240503 14:12:15.990427 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5a081ba59ad44890bf2f94874c034277
I20240503 14:12:15.990866 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet aac51b83abe14ab5b1fb5d627233e58e
I20240503 14:12:15.991135 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 74a9d58fb30d4b60ae0a44ec217fea11
I20240503 14:12:15.991868 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet c9325d5142af4b018354bf38370b61e0
I20240503 14:12:15.999362 29456 raft_consensus.cc:482] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.999588 29456 raft_consensus.cc:504] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:15.999428 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 393f27164887427d8a654c51cfc04e14 (DEFAULT_TABLE table=default.Table_b65e95f8090a40e5abb366c2aa904d60 [id=3ffa9f842d2b49b08e1c08fe6878eaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.000934 29456 leader_election.cc:288] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.001814 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:16.002068 28212 raft_consensus.cc:2454] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:16.002125 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8a07599c3614ce2ba13285c679960bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.002485 28031 catalog_manager.cc:6554] ReplaceTablet: tablet c9325d5142af4b018354bf38370b61e0 deleted and replaced by tablet 188d7b644d2842109f1e0c124903ddd9
I20240503 14:12:16.004056 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6e5d5fafa65145abbbad53d9c46f82bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.004871 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e5d5fafa65145abbbad53d9c46f82bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.005352 28030 catalog_manager.cc:6554] ReplaceTablet: tablet 5a081ba59ad44890bf2f94874c034277 deleted and replaced by tablet d859044b70fc48e8b62bf6a82ed69d9f
I20240503 14:12:16.005870 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6f397863a47c4fc3874037b9a424f901"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.002476 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.006955 28491 raft_consensus.cc:2454] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:16.006968 28290 leader_election.cc:302] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:16.007324 29456 raft_consensus.cc:2781] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.007416 29456 raft_consensus.cc:482] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.007475 29456 raft_consensus.cc:3037] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.001884 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393f27164887427d8a654c51cfc04e14. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:16.009773 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f397863a47c4fc3874037b9a424f901 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.010576 29456 raft_consensus.cc:504] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.011373 29456 leader_election.cc:288] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.012573 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2a170feff0884500b519684ac43a06b0" } modify_external_catalogs: true
I20240503 14:12:16.012943 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2a170feff0884500b519684ac43a06b0" } modify_external_catalogs: true
I20240503 14:12:16.014060 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:16.014312 28212 raft_consensus.cc:3037] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.014364 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b861dccd964092ae0f06668c9b58fe" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:16.014586 28491 raft_consensus.cc:3037] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.014058 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b3822edda94b4fd089b7e41145d1f6af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.016613 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3822edda94b4fd089b7e41145d1f6af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.017253 28212 raft_consensus.cc:2454] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:16.017674 28491 raft_consensus.cc:2454] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:16.017921 28290 leader_election.cc:302] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:16.018296 29456 raft_consensus.cc:2781] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.018280 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5bbe821329104389be37f7890f051041" } modify_external_catalogs: true
I20240503 14:12:16.018554 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5bbe821329104389be37f7890f051041" } modify_external_catalogs: true
I20240503 14:12:16.019845 28653 ts_tablet_manager.cc:1888] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:16.019953 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0982b99a02ee48f2b6379e7be56c4c02 (table default.Table_79726aaee899484aaf8147ce414cbd87 [id=8cce76d1e8a642a4b051b4c4a854be78]) successfully deleted
I20240503 14:12:16.020792 28604 tablet_bootstrap.cc:492] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.021122 28653 log.cc:1196] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/3514b79467dc42bc89fdacf8e39a000d
I20240503 14:12:16.021538 28653 ts_tablet_manager.cc:1909] T 3514b79467dc42bc89fdacf8e39a000d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.022485 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 505) took 2432 ms. Trace:
W20240503 14:12:16.022648 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c64ecffdd4a84ba5bc64aba80e1de1ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.022678 28604 tablet_bootstrap.cc:654] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.022594 28653 rpcz_store.cc:276] 0503 14:12:13.589732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.590110 (+ 378us) service_pool.cc:225] Handling call
0503 14:12:15.985993 (+2395883us) ts_tablet_manager.cc:1120] Deleting tablet 3514b79467dc42bc89fdacf8e39a000d
0503 14:12:15.986006 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.991818 (+ 5812us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.019698 (+ 27880us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.022456 (+ 2758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2395656}
I20240503 14:12:16.023082 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 506) took 2419 ms. Trace:
I20240503 14:12:16.023178 28653 rpcz_store.cc:276] 0503 14:12:13.604164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.604247 (+ 83us) service_pool.cc:225] Handling call
0503 14:12:16.022920 (+2418673us) ts_tablet_manager.cc:1120] Deleting tablet 60206e8e73e84e409ed6aba222d9ff4b
0503 14:12:16.022932 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.023061 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417341}
I20240503 14:12:16.023588 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 513) took 2280 ms. Trace:
I20240503 14:12:16.023679 28653 rpcz_store.cc:276] 0503 14:12:13.743213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.743271 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:16.023460 (+2280189us) ts_tablet_manager.cc:1120] Deleting tablet ae1cd28bc07e4ccd9097f6dee471c5c6
0503 14:12:16.023470 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.023567 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2273629}
I20240503 14:12:16.023826 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 1044f34588d045ce898435b61a90503f (table default.Table_6673b8d0a18d42c3a18f6e611c596580 [id=3c43a0fcf51143179a1ed0e1b8e333c1]) successfully deleted
I20240503 14:12:16.027920 29456 raft_consensus.cc:686] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:16.028692 29456 consensus_queue.cc:237] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.030969 28604 tablet_bootstrap.cc:492] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:16.031215 28604 ts_tablet_manager.cc:1367] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:12:16.032325 28604 raft_consensus.cc:348] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.032791 28604 raft_consensus.cc:374] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.032896 28604 raft_consensus.cc:729] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.024027 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 512) took 2281 ms. Trace:
I20240503 14:12:16.033260 28604 consensus_queue.cc:260] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.033330 28653 rpcz_store.cc:276] 0503 14:12:13.743017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.743084 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:16.023922 (+2280838us) ts_tablet_manager.cc:1120] Deleting tablet 57ab077988364bd687e54f90c3c265d5
0503 14:12:16.023931 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.024010 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2243726}
I20240503 14:12:16.033545 28661 ts_tablet_manager.cc:1888] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:16.034962 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 515) took 2288 ms. Trace:
I20240503 14:12:16.035611 28653 rpcz_store.cc:276] 0503 14:12:13.746952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.747015 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:16.034767 (+2287752us) ts_tablet_manager.cc:1120] Deleting tablet ac752d8ab4fd4a218b052f388f416494
0503 14:12:16.034782 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.034931 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254479}
I20240503 14:12:16.036480 28653 tablet_replica.cc:330] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.036769 28653 raft_consensus.cc:2229] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.037009 28653 raft_consensus.cc:2258] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.037096 28600 tablet_bootstrap.cc:492] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.037268 28600 ts_tablet_manager.cc:1367] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.137s user 0.017s sys 0.011s
I20240503 14:12:16.037796 28653 ts_tablet_manager.cc:1875] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.038146 28600 raft_consensus.cc:348] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.038360 28600 raft_consensus.cc:374] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.038416 28600 raft_consensus.cc:729] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.038470 28614 tablet_bootstrap.cc:492] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.038659 28614 ts_tablet_manager.cc:1367] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.002s
I20240503 14:12:16.038683 28600 consensus_queue.cc:260] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.039386 28600 ts_tablet_manager.cc:1398] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:16.039743 28614 raft_consensus.cc:348] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.040019 28614 raft_consensus.cc:374] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.040149 28614 raft_consensus.cc:729] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.040532 28614 consensus_queue.cc:260] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.041172 28614 ts_tablet_manager.cc:1398] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:16.041611 28614 tablet_bootstrap.cc:492] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.041884 28600 tablet_bootstrap.cc:492] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.044277 28600 tablet_bootstrap.cc:654] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.046500 28614 tablet_bootstrap.cc:654] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.047667 29595 raft_consensus.cc:482] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.047881 29595 raft_consensus.cc:504] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.048804 29595 leader_election.cc:288] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.054937 28653 ts_tablet_manager.cc:1888] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:16.055626 28653 log.cc:1196] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7945118bb17b431dac8670007d2d0a18
I20240503 14:12:16.056143 28653 ts_tablet_manager.cc:1909] T 7945118bb17b431dac8670007d2d0a18 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.058948 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 514) took 2312 ms. Trace:
W20240503 14:12:16.051569 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7b6c998e518541248a87e4c1551ec080 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b6c998e518541248a87e4c1551ec080 already in progress: creating tablet
I20240503 14:12:16.059149 28653 rpcz_store.cc:276] 0503 14:12:13.746694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.747371 (+ 677us) service_pool.cc:225] Handling call
0503 14:12:16.036417 (+2289046us) ts_tablet_manager.cc:1120] Deleting tablet 7945118bb17b431dac8670007d2d0a18
0503 14:12:16.036428 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.047608 (+ 11180us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.054796 (+ 7188us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.058919 (+ 4123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2255704}
W20240503 14:12:16.059499 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ce4846d662a64d51b16fd5447c371e9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce4846d662a64d51b16fd5447c371e9a already in progress: creating tablet
W20240503 14:12:16.059881 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 4084bd4fbf604042bb33ff03c3f8cc19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4084bd4fbf604042bb33ff03c3f8cc19 already in progress: creating tablet
I20240503 14:12:16.059955 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 511) took 2317 ms. Trace:
I20240503 14:12:16.060353 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ca6627f0d9a544bb8176bea6c1af5ab4 (table default.Table_e3e1acbcbb7c4aeab3db92cc253b441d [id=9ac543611bee46c08fb673c287aa6af8]) successfully deleted
I20240503 14:12:16.060529 28653 rpcz_store.cc:276] 0503 14:12:13.742551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.791932 (+ 49381us) service_pool.cc:225] Handling call
0503 14:12:16.059816 (+2267884us) ts_tablet_manager.cc:1120] Deleting tablet f8fe462c9ee54a27b9714f6457036d76
0503 14:12:16.059828 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.059937 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2256152}
I20240503 14:12:16.060747 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 651478ef632144079f93f3b50bff4611 (table default.Table_2c3ffc8213514499a50aa5ff0b0d6cc8 [id=ece08717161d44f7987b32a0ccc1cf36]) successfully deleted
W20240503 14:12:16.060992 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 320c526e845e4dc78da345bcb095918f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 320c526e845e4dc78da345bcb095918f already in progress: creating tablet
I20240503 14:12:16.061267 28653 tablet_replica.cc:330] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:16.061305 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a6c17cfce7e644668817bf3483a098d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c17cfce7e644668817bf3483a098d4 already in progress: creating tablet
I20240503 14:12:16.061739 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 50edc38cdbe74397aa4f4928d56d9fbc (table default.Table_9a954e776a184d819708a62a99ed94fc [id=713e339595f142ab8a82bb724d23a487]) successfully deleted
W20240503 14:12:16.062016 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 34fa067359a640eeba8c80d08371904e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34fa067359a640eeba8c80d08371904e already in progress: creating tablet
I20240503 14:12:16.062055 28653 raft_consensus.cc:2229] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.062533 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 3514b79467dc42bc89fdacf8e39a000d (table default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d [id=a0937da53798417e9c74d664681451a0]) successfully deleted
W20240503 14:12:16.062839 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 60206e8e73e84e409ed6aba222d9ff4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60206e8e73e84e409ed6aba222d9ff4b already in progress: creating tablet
I20240503 14:12:16.063020 28653 pending_rounds.cc:62] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:16.063112 28653 pending_rounds.cc:68] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597711090147328 op_type: NO_OP noop_request { }
W20240503 14:12:16.063170 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 57ab077988364bd687e54f90c3c265d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57ab077988364bd687e54f90c3c265d5 already in progress: creating tablet
W20240503 14:12:16.063442 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ac752d8ab4fd4a218b052f388f416494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac752d8ab4fd4a218b052f388f416494 already in progress: creating tablet
I20240503 14:12:16.063593 28653 raft_consensus.cc:2866] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:16.063774 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7945118bb17b431dac8670007d2d0a18 (table default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]) successfully deleted
I20240503 14:12:16.063967 28653 raft_consensus.cc:2258] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.052871 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
W20240503 14:12:16.064127 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f8fe462c9ee54a27b9714f6457036d76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8fe462c9ee54a27b9714f6457036d76 already in progress: creating tablet
I20240503 14:12:16.052124 28604 ts_tablet_manager.cc:1398] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 14:12:16.065362 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.065623 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 1603e2360041427f8635782e812b6431 with delete_type TABLET_DATA_DELETED (replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.065937 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.066192 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b52ea336524beea2f255c8720f7472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.067301 28604 tablet_bootstrap.cc:492] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.069121 28653 ts_tablet_manager.cc:1875] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.073379 29281 raft_consensus.cc:482] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.073573 29281 raft_consensus.cc:504] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.073889 28604 tablet_bootstrap.cc:654] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.073782 28032 catalog_manager.cc:5484] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:16.074328 29281 leader_election.cc:288] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.074360 28032 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:16.075358 29456 raft_consensus.cc:482] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.075515 29456 raft_consensus.cc:504] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.076853 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.076853 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.076347 29456 leader_election.cc:288] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.079811 28661 log.cc:1196] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/54f08aac9a7147ebaffda4262a57a86a
I20240503 14:12:16.080338 28661 ts_tablet_manager.cc:1909] T 54f08aac9a7147ebaffda4262a57a86a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:16.081290 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 522) took 2186 ms. Trace:
I20240503 14:12:16.081393 28661 rpcz_store.cc:276] 0503 14:12:13.894587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.912801 (+ 18214us) service_pool.cc:225] Handling call
0503 14:12:15.845702 (+1932901us) ts_tablet_manager.cc:1120] Deleting tablet 54f08aac9a7147ebaffda4262a57a86a
0503 14:12:15.845712 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:15.870728 (+ 25016us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.033423 (+162695us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.081259 (+ 47836us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1932611}
I20240503 14:12:16.082149 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:16.082424 28212 raft_consensus.cc:2379] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:16.083499 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9262a7e828489ab51a745068c737df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.083807 28603 tablet_replica.cc:330] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.084007 28603 raft_consensus.cc:2229] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.083989 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc7d96357743089372baf71654b1e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b90e9563b4c4cef8ee7964eaa96e285) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.084239 28603 raft_consensus.cc:2258] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.084365 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:16.084496 28150 leader_election.cc:341] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.084730 28150 leader_election.cc:341] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.084807 28150 leader_election.cc:302] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.084882 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet aac51b83abe14ab5b1fb5d627233e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.085132 28603 ts_tablet_manager.cc:1875] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:16.086805 28290 leader_election.cc:341] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0f4fdf88fefa4ff7939f3ca05a82d7ba
I20240503 14:12:16.087395 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d77014c66624f958b38f7cc83644891 with delete_type TABLET_DATA_DELETED (replaced by tablet 20b176a071ba4f3fbff2a64d6e46027a) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.087674 29281 raft_consensus.cc:482] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.087837 29281 raft_consensus.cc:504] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.088584 29281 leader_election.cc:288] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.088757 29281 raft_consensus.cc:2726] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.091734 28603 ts_tablet_manager.cc:1888] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.092011 28603 log.cc:1196] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ed9262a7e828489ab51a745068c737df
I20240503 14:12:16.092375 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.092470 28603 ts_tablet_manager.cc:1909] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.094388 28603 tablet_replica.cc:330] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.094606 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:16.094720 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 3703ab7db61b4c9ead3058bfdc123189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.095191 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9262a7e828489ab51a745068c737df" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.094755 28603 raft_consensus.cc:2229] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.095652 28603 raft_consensus.cc:2258] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.098111 28653 ts_tablet_manager.cc:1888] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:16.099053 28603 ts_tablet_manager.cc:1875] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.100716 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 516) took 2283 ms. Trace:
I20240503 14:12:16.100908 28661 rpcz_store.cc:276] 0503 14:12:13.817139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.836195 (+ 19056us) service_pool.cc:225] Handling call
0503 14:12:16.081694 (+2245499us) ts_tablet_manager.cc:1120] Deleting tablet 57ab077988364bd687e54f90c3c265d5
0503 14:12:16.081706 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.100688 (+ 18982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2165377}
I20240503 14:12:16.101384 28661 tablet_replica.cc:330] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:16.101665 28661 raft_consensus.cc:2229] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.098989 28290 leader_election.cc:302] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:16.101950 28290 raft_consensus.cc:2670] T ed9262a7e828489ab51a745068c737df P d3a6b9b7913444d29c96581af6febf33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:16.102394 28661 pending_rounds.cc:62] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
W20240503 14:12:16.102928 28290 leader_election.cc:341] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 0f4fdf88fefa4ff7939f3ca05a82d7ba
I20240503 14:12:16.103214 28653 log.cc:1196] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7c6f41bb9c5d40b0913264864ecc3528
I20240503 14:12:16.103691 28653 ts_tablet_manager.cc:1909] T 7c6f41bb9c5d40b0913264864ecc3528 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.103593 28661 pending_rounds.cc:68] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597706844557312 op_type: NO_OP noop_request { }
I20240503 14:12:16.103783 28661 raft_consensus.cc:2866] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:16.103857 28290 leader_election.cc:302] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.104189 29456 raft_consensus.cc:2726] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.104759 28661 raft_consensus.cc:2258] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.105150 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.105324 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 516) took 2289 ms. Trace:
I20240503 14:12:16.106098 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.107064 28600 tablet_bootstrap.cc:492] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.107913 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.107081 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.108006 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.107209 28653 rpcz_store.cc:276] 0503 14:12:13.816318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.816389 (+ 71us) service_pool.cc:225] Handling call
0503 14:12:16.061215 (+2244826us) ts_tablet_manager.cc:1120] Deleting tablet 7c6f41bb9c5d40b0913264864ecc3528
0503 14:12:16.061226 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.082757 (+ 21531us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.097964 (+ 15207us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.105295 (+ 7331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2244533}
I20240503 14:12:16.108649 28661 ts_tablet_manager.cc:1875] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.108767 28653 tablet_replica.cc:330] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.109032 28653 raft_consensus.cc:2229] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.109251 28614 tablet_bootstrap.cc:492] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.109259 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.109273 28653 raft_consensus.cc:2258] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.109413 28614 ts_tablet_manager.cc:1367] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.002s
I20240503 14:12:16.108999 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.109983 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a081ba59ad44890bf2f94874c034277 with delete_type TABLET_DATA_DELETED (replaced by tablet d859044b70fc48e8b62bf6a82ed69d9f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.110193 28614 raft_consensus.cc:348] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.110340 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.110394 28614 raft_consensus.cc:374] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.110451 28614 raft_consensus.cc:729] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.110639 28614 consensus_queue.cc:260] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.111196 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a081ba59ad44890bf2f94874c034277 with delete_type TABLET_DATA_DELETED (replaced by tablet d859044b70fc48e8b62bf6a82ed69d9f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.111276 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet aac51b83abe14ab5b1fb5d627233e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.111291 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dbb59d98834ab6bbac4abfa92681e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:04 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.111312 28614 ts_tablet_manager.cc:1398] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:16.111821 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a081ba59ad44890bf2f94874c034277 with delete_type TABLET_DATA_DELETED (replaced by tablet d859044b70fc48e8b62bf6a82ed69d9f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.111869 28614 tablet_bootstrap.cc:492] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:16.112270 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet b24ab3a5bba04a5092225a981f292222 (table default.Table_5354b912229e43539928c0e0df20a641 [id=294277efc25b401291a90d0fa70c90a9], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet b24ab3a5bba04a5092225a981f292222 consensus state has no leader
I20240503 14:12:16.112381 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet aac51b83abe14ab5b1fb5d627233e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.112052 28653 ts_tablet_manager.cc:1875] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.112808 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7945118bb17b431dac8670007d2d0a18 (table default.Table_bc29cf4457ea41e5a9d22f6d8c120f36 [id=892862b894c547e384ebf8b33b9a7ba5]) successfully deleted
I20240503 14:12:16.113740 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 54f08aac9a7147ebaffda4262a57a86a (table default.Table_fc0d4a5f37ae46d9807079682b94ba1d [id=f406220394264c8f824c0e1fce8479e1]) successfully deleted
I20240503 14:12:16.113885 28614 tablet_bootstrap.cc:654] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:16.114065 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 57ab077988364bd687e54f90c3c265d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57ab077988364bd687e54f90c3c265d5 already in progress: creating tablet
I20240503 14:12:16.114511 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ed9262a7e828489ab51a745068c737df (table default.Table_dbb71b03f52c48d29206e945b956f702 [id=fd3e3a26ac344865b4c622ebe130ebff]) successfully deleted
I20240503 14:12:16.115106 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 with delete_type TABLET_DATA_DELETED (replaced by tablet d4c8a97076a44f9eaa3a15026f90a304) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.115144 28600 ts_tablet_manager.cc:1367] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.007s
I20240503 14:12:16.116165 28600 raft_consensus.cc:348] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.118501 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:16.119300 28603 ts_tablet_manager.cc:1888] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.119474 28600 raft_consensus.cc:374] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.119612 28600 raft_consensus.cc:729] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.119938 28600 consensus_queue.cc:260] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.120287 28603 log.cc:1196] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/37dc7d96357743089372baf71654b1e3
I20240503 14:12:16.120683 28603 ts_tablet_manager.cc:1909] T 37dc7d96357743089372baf71654b1e3 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.121217 28600 ts_tablet_manager.cc:1398] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:16.121358 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1603e2360041427f8635782e812b6431" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.121034 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1603e2360041427f8635782e812b6431" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.122223 28603 tablet_replica.cc:330] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
W20240503 14:12:16.122133 28150 leader_election.cc:341] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.122779 28150 leader_election.cc:341] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.122942 28150 leader_election.cc:302] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.123312 29281 raft_consensus.cc:2726] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.124294 28600 tablet_bootstrap.cc:492] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.125469 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7c6f41bb9c5d40b0913264864ecc3528 (table default.Table_6add22bd3ca9452fa98a991f4e9b817f [id=806d4f0bc4524d9481487418ff797a5d]) successfully deleted
I20240503 14:12:16.125866 28603 raft_consensus.cc:2229] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.126127 28603 raft_consensus.cc:2258] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.127318 28600 tablet_bootstrap.cc:654] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.133623 28653 ts_tablet_manager.cc:1888] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.133872 28653 log.cc:1196] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ed9262a7e828489ab51a745068c737df
I20240503 14:12:16.134507 28653 ts_tablet_manager.cc:1909] T ed9262a7e828489ab51a745068c737df P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.134953 28603 ts_tablet_manager.cc:1875] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.135457 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 517) took 2251 ms. Trace:
I20240503 14:12:16.136240 28653 rpcz_store.cc:276] 0503 14:12:13.884387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.884601 (+ 214us) service_pool.cc:225] Handling call
0503 14:12:16.108695 (+2224094us) ts_tablet_manager.cc:1120] Deleting tablet ed9262a7e828489ab51a745068c737df
0503 14:12:16.108716 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.118552 (+ 9836us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.133493 (+ 14941us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.135425 (+ 1932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2223790}
I20240503 14:12:16.137058 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 519) took 2252 ms. Trace:
I20240503 14:12:16.137522 28653 rpcz_store.cc:276] 0503 14:12:13.885100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.885153 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:16.136911 (+2251758us) ts_tablet_manager.cc:1120] Deleting tablet 9b21cd1262d94d1c994b29c44f86cb5e
0503 14:12:16.136922 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.137035 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2251389}
I20240503 14:12:16.138356 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 518) took 2253 ms. Trace:
I20240503 14:12:16.139081 28653 rpcz_store.cc:276] 0503 14:12:13.884548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.885116 (+ 568us) service_pool.cc:225] Handling call
0503 14:12:16.138230 (+2253114us) ts_tablet_manager.cc:1120] Deleting tablet 920b41e3b21347b393497de4d4e87469
0503 14:12:16.138241 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.138340 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":338,"tablet-delete.queue_time_us":2252707}
I20240503 14:12:16.140048 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 527) took 2195 ms. Trace:
I20240503 14:12:16.140619 28653 rpcz_store.cc:276] 0503 14:12:13.944230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.944293 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:16.139912 (+2195619us) ts_tablet_manager.cc:1120] Deleting tablet aa3ce05b422a4166bfac468acb84e4cc
0503 14:12:16.139925 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.140028 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2195339}
I20240503 14:12:16.141471 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 526) took 2197 ms. Trace:
I20240503 14:12:16.141521 28604 tablet_bootstrap.cc:492] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:16.141754 28604 ts_tablet_manager.cc:1367] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.002s
I20240503 14:12:16.141922 28653 rpcz_store.cc:276] 0503 14:12:13.944018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.947786 (+ 3768us) service_pool.cc:225] Handling call
0503 14:12:16.141345 (+2193559us) ts_tablet_manager.cc:1120] Deleting tablet 6b94092725fa4956a4c00e3685f9aeaf
0503 14:12:16.141356 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.141453 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171302}
I20240503 14:12:16.142695 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 538) took 2151 ms. Trace:
I20240503 14:12:16.142617 28604 raft_consensus.cc:348] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.142869 28604 raft_consensus.cc:374] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.142958 28604 raft_consensus.cc:729] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.143218 28653 rpcz_store.cc:276] 0503 14:12:13.991319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.991769 (+ 450us) service_pool.cc:225] Handling call
0503 14:12:16.142576 (+2150807us) ts_tablet_manager.cc:1120] Deleting tablet 4fad4508a29d4822b81af6947b57e7ef
0503 14:12:16.142586 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.142679 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2142942}
I20240503 14:12:16.143190 28604 consensus_queue.cc:260] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.143796 28604 ts_tablet_manager.cc:1398] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:16.144948 28614 tablet_bootstrap.cc:492] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.145264 28614 ts_tablet_manager.cc:1367] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240503 14:12:16.146212 28614 raft_consensus.cc:348] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.146533 28614 raft_consensus.cc:374] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.146646 28614 raft_consensus.cc:729] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.146883 28604 tablet_bootstrap.cc:492] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.146958 28614 consensus_queue.cc:260] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.147166 29602 raft_consensus.cc:482] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.147465 29602 raft_consensus.cc:504] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.147650 28614 ts_tablet_manager.cc:1398] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:16.148154 28614 tablet_bootstrap.cc:492] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.148422 29602 leader_election.cc:288] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.149919 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775006e9f8f4612aa70dfb316de9ea1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.150241 28614 tablet_bootstrap.cc:654] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.150651 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775006e9f8f4612aa70dfb316de9ea1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:16.151212 28150 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.151450 28150 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.151554 28150 leader_election.cc:302] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.152012 29602 raft_consensus.cc:2726] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.144131 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 537) took 2152 ms. Trace:
I20240503 14:12:16.152864 28653 rpcz_store.cc:276] 0503 14:12:13.991175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.991719 (+ 544us) service_pool.cc:225] Handling call
0503 14:12:16.143965 (+2152246us) ts_tablet_manager.cc:1120] Deleting tablet c04707221b5241db8cd380ad5db24af5
0503 14:12:16.143975 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.144109 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2121499}
I20240503 14:12:16.153570 28653 tablet_replica.cc:330] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.154352 28600 tablet_bootstrap.cc:492] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.155066 28600 ts_tablet_manager.cc:1367] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.001s
I20240503 14:12:16.155329 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 37dc7d96357743089372baf71654b1e3 (table default.Table_85ee5bde7ad044389efb107eefd39fd3 [id=11af424ea24549bc83f6203448ae619c]) successfully deleted
W20240503 14:12:16.155699 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 31ddd791d4e14630bfb3b4b1513f76e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31ddd791d4e14630bfb3b4b1513f76e1 already in progress: creating tablet
W20240503 14:12:16.155972 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet aac51b83abe14ab5b1fb5d627233e58e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aac51b83abe14ab5b1fb5d627233e58e already in progress: creating tablet
I20240503 14:12:16.156152 28600 raft_consensus.cc:348] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.156379 28600 raft_consensus.cc:374] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.156394 28604 tablet_bootstrap.cc:654] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.156461 28600 raft_consensus.cc:729] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.156704 28600 consensus_queue.cc:260] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.157223 28600 ts_tablet_manager.cc:1398] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:16.159227 28600 tablet_bootstrap.cc:492] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.160432 28653 raft_consensus.cc:2229] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.160745 28653 raft_consensus.cc:2258] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.162889 28653 ts_tablet_manager.cc:1875] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.162940 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a1ebec4622ef4950bd1c70451958c59b
W20240503 14:12:16.162961 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69425d29d55b4fe4bb7f08ae390aaa86 consensus state has no leader
I20240503 14:12:16.165621 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a26e8389639748ae99832d7a7a9d5af3
I20240503 14:12:16.168394 28600 tablet_bootstrap.cc:654] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.173071 28264 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:16.174714 29429 raft_consensus.cc:482] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.174871 29429 raft_consensus.cc:504] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.175032 28603 ts_tablet_manager.cc:1888] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.175618 29429 leader_election.cc:288] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.175582 28603 log.cc:1196] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/1d77014c66624f958b38f7cc83644891
I20240503 14:12:16.176136 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9f5a767355f24f1b928c442018fe05c3" } new_table_name: "default.Table_13574a6ed07c45e08044a182ff0ebe82" modify_external_catalogs: true
I20240503 14:12:16.176625 28600 tablet_bootstrap.cc:492] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
W20240503 14:12:16.176826 28427 leader_election.cc:341] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: ca6627f0d9a544bb8176bea6c1af5ab4
W20240503 14:12:16.177040 28427 leader_election.cc:341] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: ca6627f0d9a544bb8176bea6c1af5ab4
I20240503 14:12:16.177134 28427 leader_election.cc:302] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:16.177248 28600 ts_tablet_manager.cc:1367] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240503 14:12:16.177938 29429 raft_consensus.cc:2726] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.178663 28603 ts_tablet_manager.cc:1909] T 1d77014c66624f958b38f7cc83644891 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.178773 28600 raft_consensus.cc:348] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.179051 28600 raft_consensus.cc:374] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.179744 28600 raft_consensus.cc:729] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.180560 28614 tablet_bootstrap.cc:492] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.180779 28614 ts_tablet_manager.cc:1367] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.004s
I20240503 14:12:16.173899 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a0610cdc38c7464888c7b7366ed72c2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.182096 28614 raft_consensus.cc:348] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.182250 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b52ef7ecedf4273939406581d843980 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:16.182408 28614 raft_consensus.cc:374] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.182643 28614 raft_consensus.cc:729] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.182941 28614 consensus_queue.cc:260] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.183574 28614 ts_tablet_manager.cc:1398] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:16.183970 28600 consensus_queue.cc:260] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.184578 28614 tablet_bootstrap.cc:492] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.187863 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1b9a58dce8ab4927bd0bd665004b2e3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.188455 28614 tablet_bootstrap.cc:654] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:16.188608 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b9a58dce8ab4927bd0bd665004b2e3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.191438 28025 catalog_manager.cc:6554] ReplaceTablet: tablet a1ebec4622ef4950bd1c70451958c59b deleted and replaced by tablet fc3f7c221cbc467dabfe5665ccd07d48
I20240503 14:12:16.191429 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ed9262a7e828489ab51a745068c737df (table default.Table_dbb71b03f52c48d29206e945b956f702 [id=fd3e3a26ac344865b4c622ebe130ebff]) successfully deleted
I20240503 14:12:16.192018 28600 ts_tablet_manager.cc:1398] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:12:16.192379 28600 tablet_bootstrap.cc:492] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.193142 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_bc29cf4457ea41e5a9d22f6d8c120f36" } new_table_name: "default.Table_4ebf150e25b946eab62caf24e5a3c292" modify_external_catalogs: true
I20240503 14:12:16.187489 28653 ts_tablet_manager.cc:1888] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:16.193691 28653 log.cc:1196] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/8c7d57fc4486457e9848daad79f04356
I20240503 14:12:16.194075 29429 raft_consensus.cc:482] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.194260 29429 raft_consensus.cc:504] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.195040 29429 leader_election.cc:288] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.195734 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_46e0d74510e14187b44fee0f210792be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.196112 28653 ts_tablet_manager.cc:1909] T 8c7d57fc4486457e9848daad79f04356 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.197535 28600 tablet_bootstrap.cc:654] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.198239 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 553) took 1445 ms. Trace:
I20240503 14:12:16.198874 28653 rpcz_store.cc:276] 0503 14:12:14.752803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.753484 (+ 681us) service_pool.cc:225] Handling call
0503 14:12:16.153515 (+1400031us) ts_tablet_manager.cc:1120] Deleting tablet 8c7d57fc4486457e9848daad79f04356
0503 14:12:16.153524 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.166323 (+ 12799us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.187339 (+ 21016us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.198219 (+ 10880us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373001}
I20240503 14:12:16.199445 28653 tablet_replica.cc:330] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:16.200181 28427 leader_election.cc:341] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 50edc38cdbe74397aa4f4928d56d9fbc
I20240503 14:12:16.200621 29456 raft_consensus.cc:482] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.200624 29598 raft_consensus.cc:482] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.200830 29456 raft_consensus.cc:504] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.200889 29598 raft_consensus.cc:504] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.201578 29598 leader_election.cc:288] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.201905 28653 raft_consensus.cc:2229] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.201615 29456 leader_election.cc:288] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.202195 28653 raft_consensus.cc:2258] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.202529 28604 tablet_bootstrap.cc:492] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:16.202860 28653 ts_tablet_manager.cc:1875] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.203466 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dbb59d98834ab6bbac4abfa92681e3" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.203660 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135b60660f448a7974520f2f87cf93a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:16.203886 28212 raft_consensus.cc:2454] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:16.203583 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135b60660f448a7974520f2f87cf93a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:16.204056 28427 leader_election.cc:341] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 50edc38cdbe74397aa4f4928d56d9fbc
I20240503 14:12:16.204234 28427 leader_election.cc:302] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:16.204489 29429 raft_consensus.cc:2726] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.202760 28604 ts_tablet_manager.cc:1367] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.007s
W20240503 14:12:16.204946 28290 leader_election.cc:341] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.205283 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0610cdc38c7464888c7b7366ed72c2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:16.206441 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9b21cd1262d94d1c994b29c44f86cb5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b21cd1262d94d1c994b29c44f86cb5e already in progress: creating tablet
I20240503 14:12:16.206511 28661 ts_tablet_manager.cc:1888] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:16.206676 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46e0d74510e14187b44fee0f210792be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.210872 28614 tablet_bootstrap.cc:492] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.211544 28614 ts_tablet_manager.cc:1367] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
W20240503 14:12:16.211968 28290 leader_election.cc:341] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a6dbb59d98834ab6bbac4abfa92681e3
I20240503 14:12:16.212203 28290 leader_election.cc:302] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.212644 28614 raft_consensus.cc:348] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.214391 29456 raft_consensus.cc:2726] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.214608 28290 leader_election.cc:302] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.214855 29456 raft_consensus.cc:2781] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.214931 29456 raft_consensus.cc:482] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.214996 29456 raft_consensus.cc:3037] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.216209 28600 tablet_bootstrap.cc:492] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.216500 28600 ts_tablet_manager.cc:1367] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
W20240503 14:12:16.216516 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 920b41e3b21347b393497de4d4e87469 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920b41e3b21347b393497de4d4e87469 already in progress: creating tablet
W20240503 14:12:16.216933 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet aa3ce05b422a4166bfac468acb84e4cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa3ce05b422a4166bfac468acb84e4cc already in progress: creating tablet
I20240503 14:12:16.217736 28600 raft_consensus.cc:348] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.218016 28600 raft_consensus.cc:374] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.218122 28600 raft_consensus.cc:729] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.218673 29456 raft_consensus.cc:504] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.218436 28600 consensus_queue.cc:260] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.219507 29456 leader_election.cc:288] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.221545 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135b60660f448a7974520f2f87cf93a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:16.221763 28212 raft_consensus.cc:3037] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.222041 28603 tablet_replica.cc:330] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.222729 28603 raft_consensus.cc:2229] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:16.223917 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6b94092725fa4956a4c00e3685f9aeaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b94092725fa4956a4c00e3685f9aeaf already in progress: creating tablet
W20240503 14:12:16.224357 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 4fad4508a29d4822b81af6947b57e7ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fad4508a29d4822b81af6947b57e7ef already in progress: creating tablet
I20240503 14:12:16.224822 28212 raft_consensus.cc:2454] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
W20240503 14:12:16.225351 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c04707221b5241db8cd380ad5db24af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c04707221b5241db8cd380ad5db24af5 already in progress: creating tablet
I20240503 14:12:16.225669 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 8c7d57fc4486457e9848daad79f04356 (table default.Table_ea4ed2f332dc46aaa1f6e7025a1a1c3c [id=5cf73f88a73a4cc5966467ecfff2140f]) successfully deleted
I20240503 14:12:16.225234 28290 leader_election.cc:302] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:16.226384 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ebec4622ef4950bd1c70451958c59b with delete_type TABLET_DATA_DELETED (replaced by tablet fc3f7c221cbc467dabfe5665ccd07d48) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.226928 29456 raft_consensus.cc:2781] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.227128 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ebec4622ef4950bd1c70451958c59b with delete_type TABLET_DATA_DELETED (replaced by tablet fc3f7c221cbc467dabfe5665ccd07d48) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.227233 28603 raft_consensus.cc:2258] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.227334 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet a26e8389639748ae99832d7a7a9d5af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.227170 29456 raft_consensus.cc:686] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:16.227598 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet dc337647c7cd402e91720478380f4696 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:16.228040 29456 consensus_queue.cc:237] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.229391 28600 ts_tablet_manager.cc:1398] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:16.229743 28600 tablet_bootstrap.cc:492] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.231756 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet a26e8389639748ae99832d7a7a9d5af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.232411 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 1d77014c66624f958b38f7cc83644891 (table default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]) successfully deleted
I20240503 14:12:16.236179 28603 ts_tablet_manager.cc:1875] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.238580 28600 tablet_bootstrap.cc:654] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.240183 28653 ts_tablet_manager.cc:1888] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.240388 28653 log.cc:1196] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0ad1cd0e80cd4adbbbbcd60e335a5c49
I20240503 14:12:16.240842 28653 ts_tablet_manager.cc:1909] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.242393 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 557) took 1462 ms. Trace:
I20240503 14:12:16.243201 28653 rpcz_store.cc:276] 0503 14:12:14.780307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.786217 (+ 5910us) service_pool.cc:225] Handling call
0503 14:12:16.199392 (+1413175us) ts_tablet_manager.cc:1120] Deleting tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49
0503 14:12:16.199402 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.212253 (+ 12851us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.240041 (+ 27788us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.242368 (+ 2327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405975}
I20240503 14:12:16.244167 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 555) took 1480 ms. Trace:
I20240503 14:12:16.244321 28653 rpcz_store.cc:276] 0503 14:12:14.763973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.776746 (+ 12773us) service_pool.cc:225] Handling call
0503 14:12:16.243992 (+1467246us) ts_tablet_manager.cc:1120] Deleting tablet c9c0540960ac45aabeaf13f888caf238
0503 14:12:16.244004 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.244146 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1425349}
I20240503 14:12:16.244956 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 556) took 1474 ms. Trace:
I20240503 14:12:16.245103 28653 rpcz_store.cc:276] 0503 14:12:14.770628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.780551 (+ 9923us) service_pool.cc:225] Handling call
0503 14:12:16.244829 (+1464278us) ts_tablet_manager.cc:1120] Deleting tablet 920be6d87c4440d0bf7453c86cfe0b5c
0503 14:12:16.244840 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.244938 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426079}
I20240503 14:12:16.245661 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 558) took 1459 ms. Trace:
I20240503 14:12:16.245831 28653 rpcz_store.cc:276] 0503 14:12:14.787886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.790227 (+ 2341us) service_pool.cc:225] Handling call
0503 14:12:16.245551 (+1455324us) ts_tablet_manager.cc:1120] Deleting tablet 77f0081cc2cb48aaa368741109032fd1
0503 14:12:16.245561 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.245644 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1424117}
I20240503 14:12:16.246553 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 567) took 1452 ms. Trace:
I20240503 14:12:16.246950 28653 rpcz_store.cc:276] 0503 14:12:14.794212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.903784 (+109572us) service_pool.cc:225] Handling call
0503 14:12:16.246442 (+1342658us) ts_tablet_manager.cc:1120] Deleting tablet 2221218847554cbd9d3242a9a6431e46
0503 14:12:16.246453 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.246536 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1323433}
I20240503 14:12:16.247727 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 568) took 1453 ms. Trace:
I20240503 14:12:16.248158 28653 rpcz_store.cc:276] 0503 14:12:14.794327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.904461 (+110134us) service_pool.cc:225] Handling call
0503 14:12:16.247577 (+1343116us) ts_tablet_manager.cc:1120] Deleting tablet ea261d8d5d174bee804aa2e171f35008
0503 14:12:16.247586 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.247690 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1289918}
I20240503 14:12:16.248944 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 577) took 1312 ms. Trace:
I20240503 14:12:16.214700 28614 raft_consensus.cc:374] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.249428 28614 raft_consensus.cc:729] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.250567 28614 consensus_queue.cc:260] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:16.251541 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6061918660e4070a0c8339cb5009ff8 consensus state has no leader
I20240503 14:12:16.251567 28653 rpcz_store.cc:276] 0503 14:12:14.936104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.957697 (+ 21593us) service_pool.cc:225] Handling call
0503 14:12:16.248824 (+1291127us) ts_tablet_manager.cc:1120] Deleting tablet 95c97a43bd714da68c1a77408f3d1cd1
0503 14:12:16.248834 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.248924 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290894}
I20240503 14:12:16.251986 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_25270ebff04b49058519a05acc90c215" } new_table_name: "default.Table_ace9146e755549ba882df639de92097f" modify_external_catalogs: true
I20240503 14:12:16.252375 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 578) took 1303 ms. Trace:
I20240503 14:12:16.252777 28614 ts_tablet_manager.cc:1398] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240503 14:12:16.252450 28653 rpcz_store.cc:276] 0503 14:12:14.948570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.957922 (+ 9352us) service_pool.cc:225] Handling call
0503 14:12:16.252233 (+1294311us) ts_tablet_manager.cc:1120] Deleting tablet ff1958dc488e4854a8c8e74da9fab243
0503 14:12:16.252243 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.252353 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294043}
I20240503 14:12:16.253639 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 579) took 1304 ms. Trace:
I20240503 14:12:16.253722 28653 rpcz_store.cc:276] 0503 14:12:14.949348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.958198 (+ 8850us) service_pool.cc:225] Handling call
0503 14:12:16.253516 (+1295318us) ts_tablet_manager.cc:1120] Deleting tablet b59945d011b64cba87c854e2ffde8226
0503 14:12:16.253526 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.253621 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295134}
W20240503 14:12:16.253866 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0993be031cc4792a9eadba9ce41d024 consensus state has no leader
I20240503 14:12:16.254123 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8fc00adb6ca146859c462157a163a3c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.254419 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c9fd8692f46d44adb6c10e08bbccdfa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.254637 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 580) took 1305 ms. Trace:
I20240503 14:12:16.254736 28653 rpcz_store.cc:276] 0503 14:12:14.949494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.958389 (+ 8895us) service_pool.cc:225] Handling call
0503 14:12:16.254517 (+1296128us) ts_tablet_manager.cc:1120] Deleting tablet f0287bc315644f0aaa976a56174576d9
0503 14:12:16.254527 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.254617 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295898}
W20240503 14:12:16.254940 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fc00adb6ca146859c462157a163a3c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:16.255017 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9fd8692f46d44adb6c10e08bbccdfa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.255168 28614 tablet_bootstrap.cc:492] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.255677 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 576) took 1333 ms. Trace:
I20240503 14:12:16.255759 28653 rpcz_store.cc:276] 0503 14:12:14.922659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.948314 (+ 25655us) service_pool.cc:225] Handling call
0503 14:12:16.255553 (+1307239us) ts_tablet_manager.cc:1120] Deleting tablet 3cbcae22ea124edaa14c9409f3052714
0503 14:12:16.255565 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.255658 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1263858}
I20240503 14:12:16.256729 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 587) took 1138 ms. Trace:
I20240503 14:12:16.256803 28653 rpcz_store.cc:276] 0503 14:12:15.118078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.118198 (+ 120us) service_pool.cc:225] Handling call
0503 14:12:16.256534 (+1138336us) ts_tablet_manager.cc:1120] Deleting tablet 2a90167ae51647688dfc502b00c5c9c6
0503 14:12:16.256544 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.256705 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136481}
I20240503 14:12:16.256979 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_556c4b7d2cad40c3b8520f4269675dc8" } new_table_name: "default.Table_bdce2849aeb843c0a01b36cd7dbcaee6" modify_external_catalogs: true
I20240503 14:12:16.257748 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 588) took 1137 ms. Trace:
I20240503 14:12:16.257818 28653 rpcz_store.cc:276] 0503 14:12:15.120453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.121044 (+ 591us) service_pool.cc:225] Handling call
0503 14:12:16.257625 (+1136581us) ts_tablet_manager.cc:1120] Deleting tablet 2cad43f3c6094253ba87d0ed3fc29fd9
0503 14:12:16.257634 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.257731 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103480}
I20240503 14:12:16.258725 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 589) took 1101 ms. Trace:
I20240503 14:12:16.258798 28653 rpcz_store.cc:276] 0503 14:12:15.156915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.157000 (+ 85us) service_pool.cc:225] Handling call
0503 14:12:16.258611 (+1101611us) ts_tablet_manager.cc:1120] Deleting tablet 5f6c2307b8e64c2481d5bd47e9189fea
0503 14:12:16.258620 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.258699 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085963}
I20240503 14:12:16.258911 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9e8bd96c6824e9c8c64036f0848c876" } modify_external_catalogs: true
I20240503 14:12:16.259135 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9e8bd96c6824e9c8c64036f0848c876" } modify_external_catalogs: true
I20240503 14:12:16.259689 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 590) took 1079 ms. Trace:
I20240503 14:12:16.259786 28653 rpcz_store.cc:276] 0503 14:12:15.180125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.180640 (+ 515us) service_pool.cc:225] Handling call
0503 14:12:16.259583 (+1078943us) ts_tablet_manager.cc:1120] Deleting tablet 9e9d878dfb144e5db914f8619c13ce8c
0503 14:12:16.259593 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.259672 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1063487}
I20240503 14:12:16.260728 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 592) took 1068 ms. Trace:
I20240503 14:12:16.260799 28653 rpcz_store.cc:276] 0503 14:12:15.200685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.200924 (+ 239us) service_pool.cc:225] Handling call
0503 14:12:16.260608 (+1059684us) ts_tablet_manager.cc:1120] Deleting tablet 77f6fb009bdd4f70ba90e7b5f972cea9
0503 14:12:16.260618 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.260701 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059383}
I20240503 14:12:16.261291 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:16.261512 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]). Expected version 2 got 0
I20240503 14:12:16.261677 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 591) took 1077 ms. Trace:
I20240503 14:12:16.261754 28653 rpcz_store.cc:276] 0503 14:12:15.184008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.191872 (+ 7864us) service_pool.cc:225] Handling call
0503 14:12:16.261571 (+1069699us) ts_tablet_manager.cc:1120] Deleting tablet d3943c8358a3484d943bad0c906a9425
0503 14:12:16.261581 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.261660 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050321}
I20240503 14:12:16.262571 28653 tablet_replica.cc:330] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.262775 28653 raft_consensus.cc:2229] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.262873 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4e0a1dcdeddb46cfac04de76adfc27e5" } modify_external_catalogs: true
I20240503 14:12:16.263141 28653 raft_consensus.cc:2258] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.263099 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4e0a1dcdeddb46cfac04de76adfc27e5" } modify_external_catalogs: true
I20240503 14:12:16.265215 28653 ts_tablet_manager.cc:1875] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.266358 28614 tablet_bootstrap.cc:654] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.268252 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (table default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]) successfully deleted
W20240503 14:12:16.268623 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c9c0540960ac45aabeaf13f888caf238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c0540960ac45aabeaf13f888caf238 already in progress: creating tablet
W20240503 14:12:16.268918 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 920be6d87c4440d0bf7453c86cfe0b5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920be6d87c4440d0bf7453c86cfe0b5c already in progress: creating tablet
W20240503 14:12:16.269290 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 77f0081cc2cb48aaa368741109032fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f0081cc2cb48aaa368741109032fd1 already in progress: creating tablet
W20240503 14:12:16.269552 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2221218847554cbd9d3242a9a6431e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2221218847554cbd9d3242a9a6431e46 already in progress: creating tablet
I20240503 14:12:16.269788 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0e680f189add4b5c96de4046321aad2d (table default.Table_2d7ecf8640f64d82908807b4c4d4fd7f [id=d1088a94cd0442c9bd9837bf9f0e1ff9]). Expected version 2 got 0
W20240503 14:12:16.269865 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ea261d8d5d174bee804aa2e171f35008 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea261d8d5d174bee804aa2e171f35008 already in progress: creating tablet
W20240503 14:12:16.270102 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 95c97a43bd714da68c1a77408f3d1cd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c97a43bd714da68c1a77408f3d1cd1 already in progress: creating tablet
W20240503 14:12:16.270361 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ff1958dc488e4854a8c8e74da9fab243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1958dc488e4854a8c8e74da9fab243 already in progress: creating tablet
W20240503 14:12:16.270602 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b59945d011b64cba87c854e2ffde8226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b59945d011b64cba87c854e2ffde8226 already in progress: creating tablet
W20240503 14:12:16.270876 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f0287bc315644f0aaa976a56174576d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0287bc315644f0aaa976a56174576d9 already in progress: creating tablet
W20240503 14:12:16.271121 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3cbcae22ea124edaa14c9409f3052714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cbcae22ea124edaa14c9409f3052714 already in progress: creating tablet
W20240503 14:12:16.271386 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2a90167ae51647688dfc502b00c5c9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a90167ae51647688dfc502b00c5c9c6 already in progress: creating tablet
I20240503 14:12:16.271545 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7fbc01371cd145eeb7b51110d14f6456" } modify_external_catalogs: true
I20240503 14:12:16.271822 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7fbc01371cd145eeb7b51110d14f6456" } modify_external_catalogs: true
W20240503 14:12:16.271651 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cad43f3c6094253ba87d0ed3fc29fd9 already in progress: creating tablet
W20240503 14:12:16.272307 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5f6c2307b8e64c2481d5bd47e9189fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f6c2307b8e64c2481d5bd47e9189fea already in progress: creating tablet
W20240503 14:12:16.272532 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9e9d878dfb144e5db914f8619c13ce8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9d878dfb144e5db914f8619c13ce8c already in progress: creating tablet
W20240503 14:12:16.272749 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f6fb009bdd4f70ba90e7b5f972cea9 already in progress: creating tablet
W20240503 14:12:16.272948 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d3943c8358a3484d943bad0c906a9425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3943c8358a3484d943bad0c906a9425 already in progress: creating tablet
I20240503 14:12:16.273283 28600 tablet_bootstrap.cc:492] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.274183 28194 tablet_service.cc:1467] Processing CreateTablet for tablet b12a2f262e324a5188ecff08c8588757 (DEFAULT_TABLE table=default.Table_cc6397f0878d4dcb939227f869da0ecf [id=0cf2a97af7b64d759283cae344f978a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.274612 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12a2f262e324a5188ecff08c8588757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.280562 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 3150dfcd21bc4604be8f9c2cbdc10648 (DEFAULT_TABLE table=default.Table_edb926a0a57b4cb090017b29fee48eab [id=b7970a12791e4dee81cfa31663476271]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.281052 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3150dfcd21bc4604be8f9c2cbdc10648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.284437 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 471992c1f7e84ef1953cd85448a492ec (DEFAULT_TABLE table=default.Table_1df0f31c455445fbac525618cff09685 [id=c68b73704a5544069ae5a0186a04dd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.284895 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471992c1f7e84ef1953cd85448a492ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.284943 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 3150dfcd21bc4604be8f9c2cbdc10648 (DEFAULT_TABLE table=default.Table_edb926a0a57b4cb090017b29fee48eab [id=b7970a12791e4dee81cfa31663476271]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.285236 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 471992c1f7e84ef1953cd85448a492ec (DEFAULT_TABLE table=default.Table_1df0f31c455445fbac525618cff09685 [id=c68b73704a5544069ae5a0186a04dd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.285404 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3150dfcd21bc4604be8f9c2cbdc10648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.285775 28330 tablet_service.cc:1467] Processing CreateTablet for tablet a4639ff7138f4c3ea6b7e5e5b4e7cd6d (DEFAULT_TABLE table=default.Table_960c4947e6bc4bb6839d7257e7d93a41 [id=6afc247a569845d0b23244ad7b58025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.286131 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4639ff7138f4c3ea6b7e5e5b4e7cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.288264 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.288738 28614 tablet_bootstrap.cc:492] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.288955 28614 ts_tablet_manager.cc:1367] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.004s
I20240503 14:12:16.289013 28325 tablet_service.cc:1467] Processing CreateTablet for tablet b12a2f262e324a5188ecff08c8588757 (DEFAULT_TABLE table=default.Table_cc6397f0878d4dcb939227f869da0ecf [id=0cf2a97af7b64d759283cae344f978a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.289413 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12a2f262e324a5188ecff08c8588757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.289374 28185 tablet_service.cc:1467] Processing CreateTablet for tablet a4639ff7138f4c3ea6b7e5e5b4e7cd6d (DEFAULT_TABLE table=default.Table_960c4947e6bc4bb6839d7257e7d93a41 [id=6afc247a569845d0b23244ad7b58025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.289748 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4639ff7138f4c3ea6b7e5e5b4e7cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.289986 28614 raft_consensus.cc:348] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.290201 28614 raft_consensus.cc:374] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.290246 28614 raft_consensus.cc:729] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.290539 28614 consensus_queue.cc:260] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.290961 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 00a56d79c36f4ffa9bae02e2369d8ded (DEFAULT_TABLE table=default.Table_34e94ee61c2644d3b8f4449f0d6f341b [id=712cd86826214f9cb96e0b815cc6f2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.291182 28614 ts_tablet_manager.cc:1398] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:16.291309 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a56d79c36f4ffa9bae02e2369d8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.291558 28614 tablet_bootstrap.cc:492] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.292420 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 00a56d79c36f4ffa9bae02e2369d8ded (DEFAULT_TABLE table=default.Table_34e94ee61c2644d3b8f4449f0d6f341b [id=712cd86826214f9cb96e0b815cc6f2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.292812 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a56d79c36f4ffa9bae02e2369d8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.293996 28614 tablet_bootstrap.cc:654] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.294992 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471992c1f7e84ef1953cd85448a492ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.296413 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 923881b2c05b4e96af9ff9dd1b92f7ec (DEFAULT_TABLE table=default.Table_75a7cf3d16ac48e69b1ee39530fedaa0 [id=65a52780fbd941b79d653b22ba769c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.296845 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923881b2c05b4e96af9ff9dd1b92f7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.300122 28661 log.cc:1196] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/3703ab7db61b4c9ead3058bfdc123189
I20240503 14:12:16.300489 28187 tablet_service.cc:1467] Processing CreateTablet for tablet d59ca5509565404480a25ec78fe7194a (DEFAULT_TABLE table=default.Table_2ef0e74fb73844b3a4d3b680c04d73e1 [id=0b746045695c4ebf935ef49cf6372b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.300695 28661 ts_tablet_manager.cc:1909] T 3703ab7db61b4c9ead3058bfdc123189 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:16.300907 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59ca5509565404480a25ec78fe7194a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.301553 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 923881b2c05b4e96af9ff9dd1b92f7ec (DEFAULT_TABLE table=default.Table_75a7cf3d16ac48e69b1ee39530fedaa0 [id=65a52780fbd941b79d653b22ba769c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.301692 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 524) took 2406 ms. Trace:
I20240503 14:12:16.301934 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923881b2c05b4e96af9ff9dd1b92f7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.301795 28661 rpcz_store.cc:276] 0503 14:12:13.894874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.917391 (+ 22517us) service_pool.cc:225] Handling call
0503 14:12:16.101325 (+2183934us) ts_tablet_manager.cc:1120] Deleting tablet 3703ab7db61b4c9ead3058bfdc123189
0503 14:12:16.101336 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.135989 (+ 34653us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.206387 (+ 70398us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.301657 (+ 95270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183601}
I20240503 14:12:16.302265 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 533) took 2345 ms. Trace:
I20240503 14:12:16.302335 28661 rpcz_store.cc:276] 0503 14:12:13.956472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.956564 (+ 92us) service_pool.cc:225] Handling call
0503 14:12:16.302105 (+2345541us) ts_tablet_manager.cc:1120] Deleting tablet aa3ce05b422a4166bfac468acb84e4cc
0503 14:12:16.302118 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.302249 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2276574}
I20240503 14:12:16.302675 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 532) took 2346 ms. Trace:
I20240503 14:12:16.302722 28324 tablet_service.cc:1467] Processing CreateTablet for tablet d59ca5509565404480a25ec78fe7194a (DEFAULT_TABLE table=default.Table_2ef0e74fb73844b3a4d3b680c04d73e1 [id=0b746045695c4ebf935ef49cf6372b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.302736 28661 rpcz_store.cc:276] 0503 14:12:13.956022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:13.956110 (+ 88us) service_pool.cc:225] Handling call
0503 14:12:16.302557 (+2346447us) ts_tablet_manager.cc:1120] Deleting tablet 6b94092725fa4956a4c00e3685f9aeaf
0503 14:12:16.302565 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.302660 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2252895}
I20240503 14:12:16.303017 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 543) took 2171 ms. Trace:
I20240503 14:12:16.303073 28661 rpcz_store.cc:276] 0503 14:12:14.131949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.133703 (+ 1754us) service_pool.cc:225] Handling call
0503 14:12:16.302920 (+2169217us) ts_tablet_manager.cc:1120] Deleting tablet 4fad4508a29d4822b81af6947b57e7ef
0503 14:12:16.302927 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.303003 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155568}
I20240503 14:12:16.303349 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 542) took 2171 ms. Trace:
I20240503 14:12:16.303411 28661 rpcz_store.cc:276] 0503 14:12:14.131841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.133668 (+ 1827us) service_pool.cc:225] Handling call
0503 14:12:16.303256 (+2169588us) ts_tablet_manager.cc:1120] Deleting tablet c04707221b5241db8cd380ad5db24af5
0503 14:12:16.303264 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.303335 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155729}
I20240503 14:12:16.303666 28661 tablet_replica.cc:330] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:16.303872 28661 raft_consensus.cc:2229] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.304034 28661 raft_consensus.cc:2258] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.208606 28604 raft_consensus.cc:348] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:16.220571 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135b60660f448a7974520f2f87cf93a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:16.232805 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet a26e8389639748ae99832d7a7a9d5af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.304227 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.304477 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d4e4a1f0b14406be957426a45bf45d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.304562 28604 raft_consensus.cc:374] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.304651 28604 raft_consensus.cc:729] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.233414 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 004d556f3ed44a7db1a52d205ecfd90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.304889 28604 consensus_queue.cc:260] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:16.233593 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6627f0d9a544bb8176bea6c1af5ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.233803 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc7d96357743089372baf71654b1e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b90e9563b4c4cef8ee7964eaa96e285) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.233944 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d77014c66624f958b38f7cc83644891 with delete_type TABLET_DATA_DELETED (replaced by tablet 20b176a071ba4f3fbff2a64d6e46027a) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.237149 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ebec4622ef4950bd1c70451958c59b with delete_type TABLET_DATA_DELETED (replaced by tablet fc3f7c221cbc467dabfe5665ccd07d48) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.237143 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 0982b99a02ee48f2b6379e7be56c4c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.307199 29602 raft_consensus.cc:482] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.307349 29602 raft_consensus.cc:504] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:16.307711 28290 leader_election.cc:341] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.308188 29602 leader_election.cc:288] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:16.242691 28427 consensus_peers.cc:485] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c7d57fc4486457e9848daad79f04356. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:16.247545 29429 raft_consensus.cc:482] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:16.308368 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 65734us
I20240503 14:12:16.308421 29429 raft_consensus.cc:504] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.309265 29429 leader_election.cc:288] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.309701 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 6371e68ee85a47cdb0e8e21f7124d94a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.310180 28458 tablet_service.cc:1467] Processing CreateTablet for tablet b12a2f262e324a5188ecff08c8588757 (DEFAULT_TABLE table=default.Table_cc6397f0878d4dcb939227f869da0ecf [id=0cf2a97af7b64d759283cae344f978a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.310627 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12a2f262e324a5188ecff08c8588757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.310885 28469 tablet_service.cc:1467] Processing CreateTablet for tablet d59ca5509565404480a25ec78fe7194a (DEFAULT_TABLE table=default.Table_2ef0e74fb73844b3a4d3b680c04d73e1 [id=0b746045695c4ebf935ef49cf6372b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.311295 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59ca5509565404480a25ec78fe7194a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.314458 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.314724 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 00a56d79c36f4ffa9bae02e2369d8ded (DEFAULT_TABLE table=default.Table_34e94ee61c2644d3b8f4449f0d6f341b [id=712cd86826214f9cb96e0b815cc6f2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.314953 28604 ts_tablet_manager.cc:1398] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.110s user 0.002s sys 0.000s
I20240503 14:12:16.315145 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a56d79c36f4ffa9bae02e2369d8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.315079 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.316408 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 471992c1f7e84ef1953cd85448a492ec (DEFAULT_TABLE table=default.Table_1df0f31c455445fbac525618cff09685 [id=c68b73704a5544069ae5a0186a04dd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.317073 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 3150dfcd21bc4604be8f9c2cbdc10648 (DEFAULT_TABLE table=default.Table_edb926a0a57b4cb090017b29fee48eab [id=b7970a12791e4dee81cfa31663476271]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:16.317281 28427 consensus_peers.cc:485] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b139ab0b5864024ac48215d46b0a7bd. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:16.317504 28427 consensus_peers.cc:485] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1044f34588d045ce898435b61a90503f. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:16.317705 28427 leader_election.cc:341] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 1d77014c66624f958b38f7cc83644891
W20240503 14:12:16.317770 28150 leader_election.cc:341] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.317991 28150 leader_election.cc:341] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.318071 28150 leader_election.cc:302] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.318738 28464 tablet_service.cc:1467] Processing CreateTablet for tablet a4639ff7138f4c3ea6b7e5e5b4e7cd6d (DEFAULT_TABLE table=default.Table_960c4947e6bc4bb6839d7257e7d93a41 [id=6afc247a569845d0b23244ad7b58025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.317466 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 923881b2c05b4e96af9ff9dd1b92f7ec (DEFAULT_TABLE table=default.Table_75a7cf3d16ac48e69b1ee39530fedaa0 [id=65a52780fbd941b79d653b22ba769c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.319031 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59ca5509565404480a25ec78fe7194a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.319128 29602 raft_consensus.cc:2726] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.321345 29602 raft_consensus.cc:482] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.321498 29602 raft_consensus.cc:504] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.322295 29602 leader_election.cc:288] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.323056 28604 tablet_bootstrap.cc:492] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.323068 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3150dfcd21bc4604be8f9c2cbdc10648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.324285 29598 raft_consensus.cc:482] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.324462 29598 raft_consensus.cc:504] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.327317 28604 tablet_bootstrap.cc:654] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.327615 29598 leader_election.cc:288] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.328488 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ba571a81eb4240a4a4a4a08cb879c5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:16.329946 28427 leader_election.cc:341] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 1d77014c66624f958b38f7cc83644891
I20240503 14:12:16.330065 28427 leader_election.cc:302] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:16.332947 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 3703ab7db61b4c9ead3058bfdc123189 (table default.Table_fb279b38e7764d3bac7f2aa1667ca58e [id=19a9b36e090c46a483c355f0f5cd6c43]) successfully deleted
W20240503 14:12:16.333276 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet aa3ce05b422a4166bfac468acb84e4cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa3ce05b422a4166bfac468acb84e4cc already in progress: creating tablet
W20240503 14:12:16.333582 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6b94092725fa4956a4c00e3685f9aeaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b94092725fa4956a4c00e3685f9aeaf already in progress: creating tablet
W20240503 14:12:16.333824 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 4fad4508a29d4822b81af6947b57e7ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fad4508a29d4822b81af6947b57e7ef already in progress: creating tablet
W20240503 14:12:16.334101 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c04707221b5241db8cd380ad5db24af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c04707221b5241db8cd380ad5db24af5 already in progress: creating tablet
I20240503 14:12:16.336071 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet dc337647c7cd402e91720478380f4696
I20240503 14:12:16.336760 28028 catalog_manager.cc:6554] ReplaceTablet: tablet dc337647c7cd402e91720478380f4696 deleted and replaced by tablet 03700c1efc08471cb67dfe3a8016a740
W20240503 14:12:16.337760 28150 leader_election.cc:341] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.338553 28028 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 7218cefb13ea4983b2111ed5fd0c997b (table default.Table_e3c825867cf84b72bfb708fcc39f6acb [id=3a24797cc53b439da44cc77025fa9f92]). Expected version 8 got 0
I20240503 14:12:16.335861 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ba571a81eb4240a4a4a4a08cb879c5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.342451 28357 raft_consensus.cc:2454] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:16.342924 28150 leader_election.cc:302] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.344200 29602 raft_consensus.cc:2781] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.344296 29602 raft_consensus.cc:482] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.344355 29602 raft_consensus.cc:3037] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.347085 29602 raft_consensus.cc:504] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.348301 29602 leader_election.cc:288] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.349287 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ba571a81eb4240a4a4a4a08cb879c5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:16.350528 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ba571a81eb4240a4a4a4a08cb879c5" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:16.350754 28357 raft_consensus.cc:3037] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:16.351217 28150 leader_election.cc:341] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.352557 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923881b2c05b4e96af9ff9dd1b92f7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.356374 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471992c1f7e84ef1953cd85448a492ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.358840 28357 raft_consensus.cc:2454] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:16.360610 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4639ff7138f4c3ea6b7e5e5b4e7cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.364275 29516 raft_consensus.cc:2726] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.365890 28150 leader_election.cc:302] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.368903 28600 ts_tablet_manager.cc:1367] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.139s user 0.008s sys 0.006s
I20240503 14:12:16.369971 28600 raft_consensus.cc:348] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.370340 28600 raft_consensus.cc:374] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.370461 28600 raft_consensus.cc:729] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.370766 28600 consensus_queue.cc:260] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.371268 29281 raft_consensus.cc:2781] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.381717 29598 raft_consensus.cc:482] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:16.382537 28290 leader_election.cc:341] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 3703ab7db61b4c9ead3058bfdc123189
W20240503 14:12:16.382870 28290 leader_election.cc:341] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 3703ab7db61b4c9ead3058bfdc123189
W20240503 14:12:16.384356 28427 consensus_peers.cc:485] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b139ab0b5864024ac48215d46b0a7bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:16.381902 29598 raft_consensus.cc:504] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.388727 29598 leader_election.cc:288] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.383041 28290 leader_election.cc:302] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.389216 29598 raft_consensus.cc:2726] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.390022 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:16.390202 28603 ts_tablet_manager.cc:1888] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.390266 28212 raft_consensus.cc:2454] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:16.390316 28603 log.cc:1196] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0f4fdf88fefa4ff7939f3ca05a82d7ba
I20240503 14:12:16.390357 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.390619 28603 ts_tablet_manager.cc:1909] T 0f4fdf88fefa4ff7939f3ca05a82d7ba P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:16.391113 28290 leader_election.cc:341] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.391202 28603 tablet_replica.cc:330] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.391484 28290 leader_election.cc:302] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.391588 28603 raft_consensus.cc:2229] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.391767 28603 raft_consensus.cc:2258] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.392376 28603 ts_tablet_manager.cc:1875] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.394649 28661 ts_tablet_manager.cc:1875] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.397898 29598 raft_consensus.cc:2781] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.398010 29598 raft_consensus.cc:482] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.398077 29598 raft_consensus.cc:3037] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.400168 28614 tablet_bootstrap.cc:492] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.400497 28614 ts_tablet_manager.cc:1367] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.013s
I20240503 14:12:16.400914 29598 raft_consensus.cc:504] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.401543 28614 raft_consensus.cc:348] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:16.401916 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 67572.7us
I20240503 14:12:16.402151 28614 raft_consensus.cc:374] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.402297 28614 raft_consensus.cc:729] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.402313 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:16.401684 29598 leader_election.cc:288] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.402520 28212 raft_consensus.cc:3037] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.402423 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:16.402688 28614 consensus_queue.cc:260] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.402945 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet dc337647c7cd402e91720478380f4696 with delete_type TABLET_DATA_DELETED (replaced by tablet 03700c1efc08471cb67dfe3a8016a740) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.403163 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.403228 28614 ts_tablet_manager.cc:1398] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240503 14:12:16.403473 28290 leader_election.cc:341] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.403530 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet e135b60660f448a7974520f2f87cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.403923 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.405247 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0f4fdf88fefa4ff7939f3ca05a82d7ba (table default.Table_a888de5f6a40467db886c3141d2a87cc [id=caca4dbaa701453794ac56475f46704e]) successfully deleted
I20240503 14:12:16.405341 28212 raft_consensus.cc:2454] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:16.405745 28290 leader_election.cc:302] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.407390 28604 tablet_bootstrap.cc:492] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:16.407544 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet dc337647c7cd402e91720478380f4696 with delete_type TABLET_DATA_DELETED (replaced by tablet 03700c1efc08471cb67dfe3a8016a740) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.407613 28604 ts_tablet_manager.cc:1367] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.085s user 0.001s sys 0.003s
I20240503 14:12:16.408188 29456 raft_consensus.cc:2781] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.408334 29456 raft_consensus.cc:686] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:16.408581 28604 raft_consensus.cc:348] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.408826 28604 raft_consensus.cc:374] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.408882 28604 raft_consensus.cc:729] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.409122 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet a7295c2d3bd34e20a69986aadc9c7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.409368 28604 consensus_queue.cc:260] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.407768 29281 raft_consensus.cc:686] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:16.409811 29281 consensus_queue.cc:237] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.409307 28600 ts_tablet_manager.cc:1398] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20240503 14:12:16.411317 28600 tablet_bootstrap.cc:492] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.413273 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet dc337647c7cd402e91720478380f4696 with delete_type TABLET_DATA_DELETED (replaced by tablet 03700c1efc08471cb67dfe3a8016a740) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.414587 28604 ts_tablet_manager.cc:1398] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:16.414906 28604 tablet_bootstrap.cc:492] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.415199 29615 raft_consensus.cc:482] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.415421 29615 raft_consensus.cc:504] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.416980 28604 tablet_bootstrap.cc:654] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.418884 28653 ts_tablet_manager.cc:1888] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:16.419008 28653 log.cc:1196] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/cc230b489ab8455c8cb2eba31084fddb
I20240503 14:12:16.419394 28653 ts_tablet_manager.cc:1909] T cc230b489ab8455c8cb2eba31084fddb P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.419783 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 593) took 1141 ms. Trace:
I20240503 14:12:16.419868 28653 rpcz_store.cc:276] 0503 14:12:15.278660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.278719 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:16.262521 (+983802us) ts_tablet_manager.cc:1120] Deleting tablet cc230b489ab8455c8cb2eba31084fddb
0503 14:12:16.262529 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.268590 (+ 6061us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.418757 (+150167us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.419752 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":983589}
I20240503 14:12:16.421941 28603 ts_tablet_manager.cc:1888] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.422070 28603 log.cc:1196] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/3703ab7db61b4c9ead3058bfdc123189
I20240503 14:12:16.422370 28603 ts_tablet_manager.cc:1909] T 3703ab7db61b4c9ead3058bfdc123189 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.423781 28600 tablet_bootstrap.cc:654] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.408600 29456 consensus_queue.cc:237] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.425133 29615 leader_election.cc:288] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.426091 29615 raft_consensus.cc:482] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.426211 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25c852aa7aa4fbfafdacec43f5bc426" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.426229 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25c852aa7aa4fbfafdacec43f5bc426" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.426266 29615 raft_consensus.cc:504] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:16.426792 28150 leader_election.cc:341] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.427013 28150 leader_election.cc:341] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.427093 28150 leader_election.cc:302] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.427551 29602 raft_consensus.cc:2726] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.429291 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65853212c2404597b015ddc8ea99ee47" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:16.429863 28150 leader_election.cc:341] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.430025 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65853212c2404597b015ddc8ea99ee47" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.430125 29602 raft_consensus.cc:482] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.430335 29602 raft_consensus.cc:504] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.431083 29602 leader_election.cc:288] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:16.431558 28150 leader_election.cc:341] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.431744 28150 leader_election.cc:302] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.431761 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b52ea336524beea2f255c8720f7472" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.431862 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b52ea336524beea2f255c8720f7472" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.432068 29602 raft_consensus.cc:2726] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:16.432322 28150 leader_election.cc:341] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.432615 29615 leader_election.cc:288] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.433977 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0c7779f7a9544e17aea7135eab2ccd5f
W20240503 14:12:16.434592 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6c90869b5034128864faaf05697c230 consensus state has no leader
I20240503 14:12:16.435619 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2b52ef7ecedf4273939406581d843980
I20240503 14:12:16.436110 28033 catalog_manager.cc:6554] ReplaceTablet: tablet 2b52ef7ecedf4273939406581d843980 deleted and replaced by tablet 7bcfe8fff82a4fdd8c01a24a207a27ab
I20240503 14:12:16.436758 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet af0137d7b1034a79bf4947c7d3f3fd57
I20240503 14:12:16.437623 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2c7ddd58b18f4c80adef853298b2019e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.438335 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c7ddd58b18f4c80adef853298b2019e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.438470 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4c09a9ca5e3d4e01820ece646f161b1c" } new_table_name: "default.Table_c2b8fa4eaa2946d8b7f1bd78f114e8fe" modify_external_catalogs: true
I20240503 14:12:16.439086 29595 raft_consensus.cc:482] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.439275 29595 raft_consensus.cc:504] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.439989 29595 leader_election.cc:288] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.440013 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 64f226db2fdb47ba9472a3aa65b953c3
W20240503 14:12:16.440814 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ac7b6df20b14d1ead036cdc42174d3a consensus state has no leader
I20240503 14:12:16.440969 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ab8c3cf894c9424f9aa6acde9e86ff7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.441350 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_691ea23c5adc4b43935e43b5f3749a49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.441707 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab8c3cf894c9424f9aa6acde9e86ff7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:16.441931 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_691ea23c5adc4b43935e43b5f3749a49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.445976 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e446d24cec95467d98278762338e11d7" } modify_external_catalogs: true
I20240503 14:12:16.446094 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_89b0f6a84c3140aca776c1da1137e09f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.446291 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e446d24cec95467d98278762338e11d7" } modify_external_catalogs: true
W20240503 14:12:16.446730 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89b0f6a84c3140aca776c1da1137e09f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.448755 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d556f3ed44a7db1a52d205ecfd90b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.449014 28489 raft_consensus.cc:2454] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:16.449452 28290 leader_election.cc:302] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:16.449751 28150 leader_election.cc:341] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.449797 28290 leader_election.cc:341] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 004d556f3ed44a7db1a52d205ecfd90b
I20240503 14:12:16.449905 28150 leader_election.cc:302] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.449769 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_425624bfb64a4882893b569d3044c2c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.450407 29281 raft_consensus.cc:482] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:16.450451 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_425624bfb64a4882893b569d3044c2c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.450534 29281 raft_consensus.cc:504] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.451234 29281 leader_election.cc:288] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.451368 29281 raft_consensus.cc:2726] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.452814 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d4e4a1f0b14406be957426a45bf45d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.453030 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d4e4a1f0b14406be957426a45bf45d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.454540 29516 raft_consensus.cc:482] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.454713 29516 raft_consensus.cc:504] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:16.455050 28150 leader_election.cc:341] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.455263 28150 leader_election.cc:341] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.455336 28150 leader_election.cc:302] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.455474 29516 leader_election.cc:288] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.455729 29602 raft_consensus.cc:2726] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.456293 28603 tablet_replica.cc:330] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.456504 28603 raft_consensus.cc:2229] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.456684 28603 raft_consensus.cc:2258] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:16.456945 28427 leader_election.cc:341] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 6371e68ee85a47cdb0e8e21f7124d94a
W20240503 14:12:16.457398 28427 leader_election.cc:341] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6371e68ee85a47cdb0e8e21f7124d94a
I20240503 14:12:16.457520 28427 leader_election.cc:302] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:16.458245 28603 ts_tablet_manager.cc:1875] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.458967 28614 tablet_bootstrap.cc:492] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.460551 28614 tablet_bootstrap.cc:654] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.461153 28653 tablet_replica.cc:330] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.461481 28653 raft_consensus.cc:2229] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.461513 29516 raft_consensus.cc:2726] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.461792 28653 raft_consensus.cc:2258] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.462455 28653 ts_tablet_manager.cc:1875] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.463924 28600 tablet_bootstrap.cc:492] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.468255 29598 raft_consensus.cc:2781] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.468397 29598 raft_consensus.cc:482] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.468465 29598 raft_consensus.cc:3037] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.464190 28600 ts_tablet_manager.cc:1367] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.001s
I20240503 14:12:16.470479 28600 raft_consensus.cc:348] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.470844 28600 raft_consensus.cc:374] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.470963 28600 raft_consensus.cc:729] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.471148 29598 raft_consensus.cc:504] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.471411 28600 consensus_queue.cc:260] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.472515 28600 ts_tablet_manager.cc:1398] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:16.473206 28600 tablet_bootstrap.cc:492] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:16.473438 28290 leader_election.cc:341] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 004d556f3ed44a7db1a52d205ecfd90b
I20240503 14:12:16.482937 28603 ts_tablet_manager.cc:1888] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.484826 28603 log.cc:1196] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a7295c2d3bd34e20a69986aadc9c7017
W20240503 14:12:16.485162 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 53666.6us
I20240503 14:12:16.485594 28603 ts_tablet_manager.cc:1909] T a7295c2d3bd34e20a69986aadc9c7017 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.485945 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.486474 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.487032 28603 tablet_replica.cc:330] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.487243 28603 raft_consensus.cc:2229] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.487396 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b52ef7ecedf4273939406581d843980 with delete_type TABLET_DATA_DELETED (replaced by tablet 7bcfe8fff82a4fdd8c01a24a207a27ab) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.487460 28603 raft_consensus.cc:2258] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.487735 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.488158 28653 ts_tablet_manager.cc:1888] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.488337 28653 log.cc:1196] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e135b60660f448a7974520f2f87cf93a
I20240503 14:12:16.488888 28653 ts_tablet_manager.cc:1909] T e135b60660f448a7974520f2f87cf93a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.489562 28603 ts_tablet_manager.cc:1875] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.493345 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b52ef7ecedf4273939406581d843980 with delete_type TABLET_DATA_DELETED (replaced by tablet 7bcfe8fff82a4fdd8c01a24a207a27ab) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.493919 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 598) took 1124 ms. Trace:
I20240503 14:12:16.483126 28614 tablet_bootstrap.cc:492] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.494335 28614 ts_tablet_manager.cc:1367] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240503 14:12:16.494019 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet af0137d7b1034a79bf4947c7d3f3fd57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.494652 28653 rpcz_store.cc:276] 0503 14:12:15.369436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.369802 (+ 366us) service_pool.cc:225] Handling call
0503 14:12:16.460610 (+1090808us) ts_tablet_manager.cc:1120] Deleting tablet e135b60660f448a7974520f2f87cf93a
0503 14:12:16.460624 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.470421 (+ 9797us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.488002 (+ 17581us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.493893 (+ 5891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090574}
I20240503 14:12:16.494699 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet af0137d7b1034a79bf4947c7d3f3fd57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.495658 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f226db2fdb47ba9472a3aa65b953c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.495914 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fb79fcf30d485798c64c815e372dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.497105 28614 raft_consensus.cc:348] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.495070 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f226db2fdb47ba9472a3aa65b953c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.496215 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d9985eecda4f3194c3251216aa58ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.498031 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 602) took 1040 ms. Trace:
I20240503 14:12:16.498184 28653 rpcz_store.cc:276] 0503 14:12:15.457209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.464177 (+ 6968us) service_pool.cc:225] Handling call
0503 14:12:16.497874 (+1033697us) ts_tablet_manager.cc:1120] Deleting tablet 79d09bee02c14822aa08050eac5be181
0503 14:12:16.497887 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.498011 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1033437}
I20240503 14:12:16.498720 28653 tablet_replica.cc:330] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.498862 28614 raft_consensus.cc:374] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.499150 28653 raft_consensus.cc:2229] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.499293 28614 raft_consensus.cc:729] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.499622 28614 consensus_queue.cc:260] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.500120 28653 pending_rounds.cc:62] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:16.500219 28653 pending_rounds.cc:68] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597717137117184 op_type: NO_OP noop_request { }
I20240503 14:12:16.500829 28614 ts_tablet_manager.cc:1398] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:16.500996 28653 raft_consensus.cc:2866] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:16.501158 28653 raft_consensus.cc:2258] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.502128 28653 ts_tablet_manager.cc:1875] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.502563 28614 tablet_bootstrap.cc:492] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.507239 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet af0137d7b1034a79bf4947c7d3f3fd57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.507923 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f226db2fdb47ba9472a3aa65b953c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.508738 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b52ef7ecedf4273939406581d843980 with delete_type TABLET_DATA_DELETED (replaced by tablet 7bcfe8fff82a4fdd8c01a24a207a27ab) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.508863 29516 raft_consensus.cc:482] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.508937 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004d556f3ed44a7db1a52d205ecfd90b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:16.509019 29516 raft_consensus.cc:504] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.509169 28489 raft_consensus.cc:3037] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.509799 29516 leader_election.cc:288] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.510582 29598 leader_election.cc:288] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.511056 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76]) successfully deleted
I20240503 14:12:16.511317 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e135b60660f448a7974520f2f87cf93a (table default.Table_00e404138f154498b730db1de48ded05 [id=6edae5b6460546ae80b1f25ffec68d48]) successfully deleted
W20240503 14:12:16.511585 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 79d09bee02c14822aa08050eac5be181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79d09bee02c14822aa08050eac5be181 already in progress: creating tablet
I20240503 14:12:16.511788 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 3703ab7db61b4c9ead3058bfdc123189 (table default.Table_fb279b38e7764d3bac7f2aa1667ca58e [id=19a9b36e090c46a483c355f0f5cd6c43]) successfully deleted
I20240503 14:12:16.512414 28489 raft_consensus.cc:2454] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:16.514559 28290 leader_election.cc:302] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:16.514878 29598 raft_consensus.cc:2781] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.515018 29598 raft_consensus.cc:686] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:16.515854 29598 consensus_queue.cc:237] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:16.516748 28427 leader_election.cc:341] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0982b99a02ee48f2b6379e7be56c4c02
W20240503 14:12:16.517510 28427 leader_election.cc:341] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 0982b99a02ee48f2b6379e7be56c4c02
W20240503 14:12:16.512018 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c9325d5142af4b018354bf38370b61e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9325d5142af4b018354bf38370b61e0 already in progress: creating tablet
I20240503 14:12:16.517642 28427 leader_election.cc:302] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:16.517868 29516 raft_consensus.cc:2726] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:16.518193 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 74a9d58fb30d4b60ae0a44ec217fea11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74a9d58fb30d4b60ae0a44ec217fea11 already in progress: creating tablet
I20240503 14:12:16.518940 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a7295c2d3bd34e20a69986aadc9c7017 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]) successfully deleted
I20240503 14:12:16.521486 28653 ts_tablet_manager.cc:1888] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:16.522354 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 62e4f6e3908e420e9fa156ee24b437d6 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:16.523069 28614 tablet_bootstrap.cc:654] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.523214 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_215b8963e0d64362a80720493fdbcdd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.523269 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ff836f0ad0b34e63bf9ee17c3f678da8" } new_table_name: "default.Table_bcabacb43f9f4398934ee7578ce08ffc" modify_external_catalogs: true
W20240503 14:12:16.523900 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_215b8963e0d64362a80720493fdbcdd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.524632 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_46ee005206eb49879a8df1d11d8d331c" } new_table_name: "default.Table_a7b16b4f89d24e62b242b1f47612920d" modify_external_catalogs: true
I20240503 14:12:16.525338 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f080c95ff5ca41e0b3d27382040107e9
I20240503 14:12:16.525635 28603 ts_tablet_manager.cc:1888] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.525761 28603 log.cc:1196] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a6dbb59d98834ab6bbac4abfa92681e3
I20240503 14:12:16.526052 28603 ts_tablet_manager.cc:1909] T a6dbb59d98834ab6bbac4abfa92681e3 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.526379 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_85a010a99b4d4402a290c308cf51e85c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:16.526947 28603 tablet_replica.cc:330] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
W20240503 14:12:16.526973 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85a010a99b4d4402a290c308cf51e85c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.528044 29595 raft_consensus.cc:482] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.528191 29595 raft_consensus.cc:504] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.528370 28653 log.cc:1196] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/47ba571a81eb4240a4a4a4a08cb879c5
I20240503 14:12:16.528803 28653 ts_tablet_manager.cc:1909] T 47ba571a81eb4240a4a4a4a08cb879c5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.528838 29595 leader_election.cc:288] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.527915 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7cafc45f1ec54604b2608f91acef4847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.529781 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cafc45f1ec54604b2608f91acef4847 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.530097 29516 raft_consensus.cc:482] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.530275 29516 raft_consensus.cc:504] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.531173 29516 leader_election.cc:288] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.530510 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 604) took 1069 ms. Trace:
I20240503 14:12:16.531697 28653 rpcz_store.cc:276] 0503 14:12:15.460726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.473203 (+ 12477us) service_pool.cc:225] Handling call
0503 14:12:16.498664 (+1025461us) ts_tablet_manager.cc:1120] Deleting tablet 47ba571a81eb4240a4a4a4a08cb879c5
0503 14:12:16.498676 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.518319 (+ 19643us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.521359 (+ 3040us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.530477 (+ 9118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1024677}
I20240503 14:12:16.532833 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775006e9f8f4612aa70dfb316de9ea1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:16.533059 28212 raft_consensus.cc:2454] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:16.533234 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 603) took 1072 ms. Trace:
I20240503 14:12:16.533440 28653 rpcz_store.cc:276] 0503 14:12:15.460524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.473879 (+ 13355us) service_pool.cc:225] Handling call
0503 14:12:16.532835 (+1058956us) ts_tablet_manager.cc:1120] Deleting tablet d43823da09bb49be8f5d1a1149a4b11d
0503 14:12:16.532846 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.532961 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1058607}
I20240503 14:12:16.533619 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775006e9f8f4612aa70dfb316de9ea1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.534307 28653 tablet_replica.cc:330] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.534739 28653 raft_consensus.cc:2229] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:16.534665 28290 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.535010 28290 leader_election.cc:302] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.535383 28653 raft_consensus.cc:2258] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.536164 28653 ts_tablet_manager.cc:1875] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.540295 29456 raft_consensus.cc:2781] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.540541 29456 raft_consensus.cc:482] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.540704 29456 raft_consensus.cc:3037] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.543711 29456 raft_consensus.cc:504] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.544844 28653 ts_tablet_manager.cc:1888] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.544986 28653 log.cc:1196] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:16.545303 28653 ts_tablet_manager.cc:1909] T 5775006e9f8f4612aa70dfb316de9ea1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:16.546231 28427 leader_election.cc:341] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0ad1cd0e80cd4adbbbbcd60e335a5c49
W20240503 14:12:16.546608 28427 leader_election.cc:341] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 0ad1cd0e80cd4adbbbbcd60e335a5c49
I20240503 14:12:16.546828 28427 leader_election.cc:302] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:16.547984 29516 raft_consensus.cc:2726] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.550922 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet f080c95ff5ca41e0b3d27382040107e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.551218 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fb79fcf30d485798c64c815e372dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.551545 28653 tablet_replica.cc:330] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.551821 28653 raft_consensus.cc:2229] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.552042 28653 raft_consensus.cc:2258] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.552673 28661 ts_tablet_manager.cc:1888] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.552717 28653 ts_tablet_manager.cc:1875] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.552906 28661 log.cc:1196] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/63e977f7837e49fb93ef0fd18b619792
I20240503 14:12:16.553318 28661 ts_tablet_manager.cc:1909] T 63e977f7837e49fb93ef0fd18b619792 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:16.556648 29615 raft_consensus.cc:482] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.556808 29615 raft_consensus.cc:504] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.557498 29615 leader_election.cc:288] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.558394 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e680f189add4b5c96de4046321aad2d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.558606 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 546) took 2258 ms. Trace:
I20240503 14:12:16.557516 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet f080c95ff5ca41e0b3d27382040107e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.555714 28604 tablet_bootstrap.cc:492] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
W20240503 14:12:16.558990 28150 leader_election.cc:341] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.559118 28604 ts_tablet_manager.cc:1367] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.144s user 0.000s sys 0.004s
I20240503 14:12:16.560617 28604 raft_consensus.cc:348] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.560992 28604 raft_consensus.cc:374] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.560988 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 47ba571a81eb4240a4a4a4a08cb879c5 (table default.Table_852b77a0ef2e4ee6bda7be721348aff8 [id=0a16a4e85e1842f1a2880d2873d363b1]) successfully deleted
I20240503 14:12:16.561129 28604 raft_consensus.cc:729] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:16.561399 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d43823da09bb49be8f5d1a1149a4b11d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43823da09bb49be8f5d1a1149a4b11d already in progress: creating tablet
I20240503 14:12:16.561461 28604 consensus_queue.cc:260] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.561684 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 5775006e9f8f4612aa70dfb316de9ea1 (table default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842]) successfully deleted
I20240503 14:12:16.563910 28604 ts_tablet_manager.cc:1398] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:16.564337 28604 tablet_bootstrap.cc:492] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.561964 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a6dbb59d98834ab6bbac4abfa92681e3 (table default.Table_33f62889c4784d6cb42ac76faa20113c [id=c62152292eb14f4aaae391155d12da6f]) successfully deleted
W20240503 14:12:16.569734 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5a081ba59ad44890bf2f94874c034277 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a081ba59ad44890bf2f94874c034277 already in progress: creating tablet
I20240503 14:12:16.571291 28653 ts_tablet_manager.cc:1888] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.571432 28653 log.cc:1196] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/1603e2360041427f8635782e812b6431
I20240503 14:12:16.571733 28653 ts_tablet_manager.cc:1909] T 1603e2360041427f8635782e812b6431 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.571513 29575 log.cc:824] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:16.573756 28600 tablet_bootstrap.cc:654] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:16.576239 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a1ebec4622ef4950bd1c70451958c59b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ebec4622ef4950bd1c70451958c59b already in progress: creating tablet
W20240503 14:12:16.576668 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a26e8389639748ae99832d7a7a9d5af3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a26e8389639748ae99832d7a7a9d5af3 already in progress: creating tablet
I20240503 14:12:16.578406 28653 tablet_replica.cc:330] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.578699 28653 raft_consensus.cc:2229] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.578913 28653 raft_consensus.cc:2258] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.579535 28653 ts_tablet_manager.cc:1875] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:16.585922 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b435cb73d3e4c9fbeadaf0d54b2bd4f consensus state has no leader
I20240503 14:12:16.587621 28653 ts_tablet_manager.cc:1888] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.587888 28653 log.cc:1196] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/00b52ea336524beea2f255c8720f7472
I20240503 14:12:16.588313 28653 ts_tablet_manager.cc:1909] T 00b52ea336524beea2f255c8720f7472 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.589071 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 1603e2360041427f8635782e812b6431 (table default.Table_cf24dd3520bd4373acfd719f2c7737ad [id=e0a3f887fca8477ba19c4ed3738b1955]) successfully deleted
I20240503 14:12:16.589466 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 1292a339f00247369ebeaa35752d51f8 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:16.589486 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 31ddd791d4e14630bfb3b4b1513f76e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31ddd791d4e14630bfb3b4b1513f76e1 already in progress: creating tablet
W20240503 14:12:16.589933 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4b2b486f56f45cca8f015deaf61150a consensus state has no leader
I20240503 14:12:16.590672 28653 tablet_replica.cc:330] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.590942 28653 raft_consensus.cc:2229] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.591167 28653 raft_consensus.cc:2258] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.591766 28653 ts_tablet_manager.cc:1875] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.593276 28600 tablet_bootstrap.cc:492] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.593557 28600 ts_tablet_manager.cc:1367] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.120s user 0.032s sys 0.000s
I20240503 14:12:16.594864 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 00b52ea336524beea2f255c8720f7472 (table default.Table_9bcbc7e8d1154ffd94951feb50445bc2 [id=eb3d80381dc844dcac4ed2bc171565d0]) successfully deleted
W20240503 14:12:16.595185 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 74a9d58fb30d4b60ae0a44ec217fea11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74a9d58fb30d4b60ae0a44ec217fea11 already in progress: creating tablet
W20240503 14:12:16.595440 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c9325d5142af4b018354bf38370b61e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9325d5142af4b018354bf38370b61e0 already in progress: creating tablet
W20240503 14:12:16.595700 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5a081ba59ad44890bf2f94874c034277 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a081ba59ad44890bf2f94874c034277 already in progress: creating tablet
W20240503 14:12:16.595978 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet aac51b83abe14ab5b1fb5d627233e58e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aac51b83abe14ab5b1fb5d627233e58e already in progress: creating tablet
I20240503 14:12:16.602617 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]). Expected version 2 got 0
I20240503 14:12:16.597043 28600 raft_consensus.cc:348] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.604033 28600 raft_consensus.cc:374] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.604135 28600 raft_consensus.cc:729] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.598582 28653 ts_tablet_manager.cc:1888] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.604414 28653 log.cc:1196] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/8b1728b6f0e64661aaa8184ce51366d2
I20240503 14:12:16.604723 28653 ts_tablet_manager.cc:1909] T 8b1728b6f0e64661aaa8184ce51366d2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.605652 28604 tablet_bootstrap.cc:654] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.606002 28653 tablet_replica.cc:330] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.606197 28653 raft_consensus.cc:2229] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.606127 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 8b1728b6f0e64661aaa8184ce51366d2 (table default.Table_80d78826082349c882ea2c0ca66e0141 [id=bc70f87f0df24f569c3cc1c65a86f667]) successfully deleted
I20240503 14:12:16.606333 28653 raft_consensus.cc:2258] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.606606 28600 consensus_queue.cc:260] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:16.606830 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a1ebec4622ef4950bd1c70451958c59b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ebec4622ef4950bd1c70451958c59b already in progress: creating tablet
W20240503 14:12:16.608341 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a26e8389639748ae99832d7a7a9d5af3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a26e8389639748ae99832d7a7a9d5af3 already in progress: creating tablet
I20240503 14:12:16.609814 28653 ts_tablet_manager.cc:1875] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.611904 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet daf5d153ab5142c0902cc6e96629f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.612612 28600 ts_tablet_manager.cc:1398] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 14:12:16.549371 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet f080c95ff5ca41e0b3d27382040107e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.553648 28614 tablet_bootstrap.cc:492] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.560441 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e680f189add4b5c96de4046321aad2d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.566592 28603 raft_consensus.cc:2229] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.612987 28614 ts_tablet_manager.cc:1367] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.004s
I20240503 14:12:16.613408 28603 pending_rounds.cc:62] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:16.613466 28603 pending_rounds.cc:68] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597717574492160 op_type: NO_OP noop_request { }
I20240503 14:12:16.613585 28603 raft_consensus.cc:2866] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:16.613667 28603 raft_consensus.cc:2258] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.613703 28614 raft_consensus.cc:348] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.613898 28614 raft_consensus.cc:374] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.613945 28614 raft_consensus.cc:729] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
W20240503 14:12:16.613931 28150 leader_election.cc:341] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.614068 28150 leader_election.cc:302] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.614166 28614 consensus_queue.cc:260] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.614403 29615 raft_consensus.cc:2726] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.614748 28614 ts_tablet_manager.cc:1398] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:16.615031 28614 tablet_bootstrap.cc:492] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.618208 28600 tablet_bootstrap.cc:492] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.619593 28600 tablet_bootstrap.cc:654] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.619689 28603 ts_tablet_manager.cc:1875] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.629798 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 07354861ee6142d6b1fbe573397d3160
I20240503 14:12:16.630932 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 07354861ee6142d6b1fbe573397d3160 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.630990 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 07354861ee6142d6b1fbe573397d3160 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.634271 28614 tablet_bootstrap.cc:654] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.622409 28653 ts_tablet_manager.cc:1888] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.635563 28653 log.cc:1196] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/b25c852aa7aa4fbfafdacec43f5bc426
I20240503 14:12:16.635769 28600 tablet_bootstrap.cc:492] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.636147 28600 ts_tablet_manager.cc:1367] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 14:12:16.636991 28653 ts_tablet_manager.cc:1909] T b25c852aa7aa4fbfafdacec43f5bc426 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.637953 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2f39f2002afd4906a9f03f5dcce7e694" } modify_external_catalogs: true
I20240503 14:12:16.638243 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2f39f2002afd4906a9f03f5dcce7e694" } modify_external_catalogs: true
I20240503 14:12:16.638656 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet b25c852aa7aa4fbfafdacec43f5bc426 (table default.Table_3e4ec426791c4b11b43d3c36260eedd4 [id=c14f7d47ddd64652aa553945f775aec6]) successfully deleted
I20240503 14:12:16.640585 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_85ee5bde7ad044389efb107eefd39fd3" } new_table_name: "default.Table_4d12dbb4326d40ddbeb81be96c45901b" modify_external_catalogs: true
I20240503 14:12:16.641048 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a52e2109f2f847dc992dd85cf261b6e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.641762 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a52e2109f2f847dc992dd85cf261b6e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.642242 28600 raft_consensus.cc:348] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.642555 28600 raft_consensus.cc:374] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.642652 28600 raft_consensus.cc:729] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.642971 28600 consensus_queue.cc:260] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.644364 28600 ts_tablet_manager.cc:1398] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:16.645546 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]). Expected version 2 got 0
I20240503 14:12:16.645807 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:16.646631 28653 tablet_replica.cc:330] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.647982 29456 leader_election.cc:288] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.647848 28653 raft_consensus.cc:2229] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.648391 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5775006e9f8f4612aa70dfb316de9ea1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:16.648432 28661 rpcz_store.cc:276] 0503 14:12:14.301888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.302156 (+ 268us) service_pool.cc:225] Handling call
0503 14:12:16.303602 (+2001446us) ts_tablet_manager.cc:1120] Deleting tablet 63e977f7837e49fb93ef0fd18b619792
0503 14:12:16.303611 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.397979 (+ 94368us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.552541 (+154562us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.558582 (+ 6041us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1998672}
I20240503 14:12:16.648813 28661 tablet_replica.cc:330] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:16.649104 28653 raft_consensus.cc:2258] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.650373 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_13574a6ed07c45e08044a182ff0ebe82" } modify_external_catalogs: true
I20240503 14:12:16.649322 28028 catalog_manager.cc:5484] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:16.650617 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_13574a6ed07c45e08044a182ff0ebe82" } modify_external_catalogs: true
I20240503 14:12:16.650841 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]). Expected version 2 got 0
I20240503 14:12:16.653308 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 63e977f7837e49fb93ef0fd18b619792 (table default.Table_c5a6b4d2127b4a1eac85c1bba95d40a4 [id=dcdb4c8fe4774779b0303db262dbfcc8]) successfully deleted
W20240503 14:12:16.651101 28290 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:16.654408 28290 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:16.654523 28290 leader_election.cc:302] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.654371 28653 ts_tablet_manager.cc:1875] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.654769 29598 raft_consensus.cc:2726] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.657114 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2331aec871824a238f3f64de217d7092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.652786 29516 raft_consensus.cc:482] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.657281 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0f75e102dd4249c3800e1fcb4339acf3
I20240503 14:12:16.653242 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 07354861ee6142d6b1fbe573397d3160 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.657459 29516 raft_consensus.cc:504] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.657912 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:16.658243 29516 leader_election.cc:288] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.658370 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 62e4f6e3908e420e9fa156ee24b437d6 deleted and replaced by tablet 843f0b87687340558d7c6acbbad2c4d7
I20240503 14:12:16.659180 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.659299 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1292a339f00247369ebeaa35752d51f8
I20240503 14:12:16.659157 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f75e102dd4249c3800e1fcb4339acf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.659854 28033 catalog_manager.cc:6554] ReplaceTablet: tablet 1292a339f00247369ebeaa35752d51f8 deleted and replaced by tablet 2ba0bac965af4446b601ff70f474b3ab
I20240503 14:12:16.660076 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 1292a339f00247369ebeaa35752d51f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba0bac965af4446b601ff70f474b3ab) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.659454 28661 raft_consensus.cc:2229] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.660543 28661 pending_rounds.cc:62] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:16.660631 28661 pending_rounds.cc:68] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597711770828800 op_type: NO_OP noop_request { }
I20240503 14:12:16.660866 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f75e102dd4249c3800e1fcb4339acf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.660953 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.660765 28661 raft_consensus.cc:2866] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:16.661154 28661 raft_consensus.cc:2258] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.661262 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1292a339f00247369ebeaa35752d51f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba0bac965af4446b601ff70f474b3ab) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.661825 28661 ts_tablet_manager.cc:1875] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.663926 28600 tablet_bootstrap.cc:492] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.664855 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 6615052ddbde487e840b6fd77a140e58 (DEFAULT_TABLE table=default.Table_46e0d74510e14187b44fee0f210792be [id=5273494231d14abe840a855258cad187]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.665284 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6615052ddbde487e840b6fd77a140e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.665689 28600 tablet_bootstrap.cc:654] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.666131 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 6615052ddbde487e840b6fd77a140e58 (DEFAULT_TABLE table=default.Table_46e0d74510e14187b44fee0f210792be [id=5273494231d14abe840a855258cad187]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.666515 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6615052ddbde487e840b6fd77a140e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.668784 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 6eab5a52d6354bcdbd507741f06f21cf (DEFAULT_TABLE table=default.Table_a0610cdc38c7464888c7b7366ed72c2f [id=689aee2341af4eee99e6f8548d2348ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.669745 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eab5a52d6354bcdbd507741f06f21cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.672634 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 6eab5a52d6354bcdbd507741f06f21cf (DEFAULT_TABLE table=default.Table_a0610cdc38c7464888c7b7366ed72c2f [id=689aee2341af4eee99e6f8548d2348ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.673038 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eab5a52d6354bcdbd507741f06f21cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.682885 28603 ts_tablet_manager.cc:1888] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:16.683127 28603 log.cc:1196] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/004d556f3ed44a7db1a52d205ecfd90b
I20240503 14:12:16.683581 28603 ts_tablet_manager.cc:1909] T 004d556f3ed44a7db1a52d205ecfd90b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.684464 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 004d556f3ed44a7db1a52d205ecfd90b (table default.Table_5264956c580441a6a23c96d39a41ea71 [id=59887894cf0843df9cd01850035c1e64]) successfully deleted
I20240503 14:12:16.686290 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240abf8c2b2343408fc98a8dac1fdf09" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:16.686512 28215 raft_consensus.cc:2454] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:16.687187 28600 tablet_bootstrap.cc:492] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.687683 28600 ts_tablet_manager.cc:1367] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20240503 14:12:16.688028 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f75e102dd4249c3800e1fcb4339acf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.688540 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240abf8c2b2343408fc98a8dac1fdf09" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.688715 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 1292a339f00247369ebeaa35752d51f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba0bac965af4446b601ff70f474b3ab) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.688755 28352 raft_consensus.cc:2454] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:16.689069 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.689322 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 6615052ddbde487e840b6fd77a140e58 (DEFAULT_TABLE table=default.Table_46e0d74510e14187b44fee0f210792be [id=5273494231d14abe840a855258cad187]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.689772 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6615052ddbde487e840b6fd77a140e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.692885 28336 tablet_service.cc:1467] Processing CreateTablet for tablet dc866a7b369147beb135b2728892b4e2 (DEFAULT_TABLE table=default.Table_6f397863a47c4fc3874037b9a424f901 [id=670af96ae0484bb897c5647d09adb2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.693439 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc866a7b369147beb135b2728892b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.694003 28614 tablet_bootstrap.cc:492] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.694274 28614 ts_tablet_manager.cc:1367] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.000s
I20240503 14:12:16.688521 28600 raft_consensus.cc:348] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.694870 28600 raft_consensus.cc:374] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.694994 28600 raft_consensus.cc:729] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.695392 28600 consensus_queue.cc:260] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.695626 28614 raft_consensus.cc:348] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.695964 28614 raft_consensus.cc:374] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.696139 28614 raft_consensus.cc:729] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.696444 28187 tablet_service.cc:1467] Processing CreateTablet for tablet dc866a7b369147beb135b2728892b4e2 (DEFAULT_TABLE table=default.Table_6f397863a47c4fc3874037b9a424f901 [id=670af96ae0484bb897c5647d09adb2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.696558 28614 consensus_queue.cc:260] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.697764 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc866a7b369147beb135b2728892b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.696842 28603 tablet_replica.cc:330] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.697888 28614 ts_tablet_manager.cc:1398] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.698369 28603 raft_consensus.cc:2229] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.699970 28614 tablet_bootstrap.cc:492] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.700227 28603 raft_consensus.cc:2258] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.702317 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 310abf35ae02497c9d3696a0c49774ae (DEFAULT_TABLE table=default.Table_c8a07599c3614ce2ba13285c679960bb [id=7dec795bfb594963aab809f41479a05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.703660 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310abf35ae02497c9d3696a0c49774ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.703943 28603 ts_tablet_manager.cc:1875] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.704607 28600 ts_tablet_manager.cc:1398] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 14:12:16.705047 28600 tablet_bootstrap.cc:492] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.706588 28600 tablet_bootstrap.cc:654] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.706729 28614 tablet_bootstrap.cc:654] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.708479 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 310abf35ae02497c9d3696a0c49774ae (DEFAULT_TABLE table=default.Table_c8a07599c3614ce2ba13285c679960bb [id=7dec795bfb594963aab809f41479a05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.708983 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310abf35ae02497c9d3696a0c49774ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.712801 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 0f642b306a8c49fd9c070a9d340889a2 (DEFAULT_TABLE table=default.Table_b3822edda94b4fd089b7e41145d1f6af [id=c631de85e8d14c29b2ec6a13ac00d56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.713264 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f642b306a8c49fd9c070a9d340889a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.714161 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 188d7b644d2842109f1e0c124903ddd9 (DEFAULT_TABLE table=default.Table_2ce3b6f2b77b4607856fc2a64ead5bb5 [id=db393a7e2a83466385ffc0af3d2789c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.714210 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 0f642b306a8c49fd9c070a9d340889a2 (DEFAULT_TABLE table=default.Table_b3822edda94b4fd089b7e41145d1f6af [id=c631de85e8d14c29b2ec6a13ac00d56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.714536 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188d7b644d2842109f1e0c124903ddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.716460 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 (DEFAULT_TABLE table=default.Table_1b9a58dce8ab4927bd0bd665004b2e3b [id=0d0ab0d2fe1b4c0ea6e23c4122255dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.716833 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281667db9a9c43d3b9d7dd9ee14b4b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.716962 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 (DEFAULT_TABLE table=default.Table_1b9a58dce8ab4927bd0bd665004b2e3b [id=0d0ab0d2fe1b4c0ea6e23c4122255dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.720436 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 188d7b644d2842109f1e0c124903ddd9 (DEFAULT_TABLE table=default.Table_2ce3b6f2b77b4607856fc2a64ead5bb5 [id=db393a7e2a83466385ffc0af3d2789c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.720844 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188d7b644d2842109f1e0c124903ddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.721024 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281667db9a9c43d3b9d7dd9ee14b4b06. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:16.724865 28550 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.DeleteTable -> {remote=127.27.50.252:45363, user_credentials={real_user=slave}} blocked reactor thread for 53914us
I20240503 14:12:16.725297 28185 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f7c221cbc467dabfe5665ccd07d48 (DEFAULT_TABLE table=default.Table_fc189ad8c19041f7814fac6de9f482cb [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.725697 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f7c221cbc467dabfe5665ccd07d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.727618 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f642b306a8c49fd9c070a9d340889a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.735678 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:16.737416 28653 ts_tablet_manager.cc:1888] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.739154 28653 log.cc:1196] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/b7d4e4a1f0b14406be957426a45bf45d
I20240503 14:12:16.743928 28653 ts_tablet_manager.cc:1909] T b7d4e4a1f0b14406be957426a45bf45d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.744786 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 6eab5a52d6354bcdbd507741f06f21cf (DEFAULT_TABLE table=default.Table_a0610cdc38c7464888c7b7366ed72c2f [id=689aee2341af4eee99e6f8548d2348ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.745417 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eab5a52d6354bcdbd507741f06f21cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.750980 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_193453f3dc0a440db9aaeada83e07e1a" } modify_external_catalogs: true
I20240503 14:12:16.751295 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_193453f3dc0a440db9aaeada83e07e1a" } modify_external_catalogs: true
I20240503 14:12:16.751888 29429 raft_consensus.cc:482] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.752063 29429 raft_consensus.cc:504] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:16.752724 28471 tablet_service.cc:1467] Processing CreateTablet for tablet dc866a7b369147beb135b2728892b4e2 (DEFAULT_TABLE table=default.Table_6f397863a47c4fc3874037b9a424f901 [id=670af96ae0484bb897c5647d09adb2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.752785 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 310abf35ae02497c9d3696a0c49774ae (DEFAULT_TABLE table=default.Table_c8a07599c3614ce2ba13285c679960bb [id=7dec795bfb594963aab809f41479a05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.753131 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc866a7b369147beb135b2728892b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.753358 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310abf35ae02497c9d3696a0c49774ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.753793 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_df59445155274498a94afe77d8f56c14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.754451 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df59445155274498a94afe77d8f56c14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.754938 28464 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f7c221cbc467dabfe5665ccd07d48 (DEFAULT_TABLE table=default.Table_fc189ad8c19041f7814fac6de9f482cb [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.755319 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f7c221cbc467dabfe5665ccd07d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.755957 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 5750311061fa4a62a9fed5086fa3e324 (DEFAULT_TABLE table=default.Table_f9806bd6f25a48bfb0a4a65437072534 [id=879354530b66475384a81c299a33d896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.756367 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5750311061fa4a62a9fed5086fa3e324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.756383 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 0f642b306a8c49fd9c070a9d340889a2 (DEFAULT_TABLE table=default.Table_b3822edda94b4fd089b7e41145d1f6af [id=c631de85e8d14c29b2ec6a13ac00d56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.756691 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f642b306a8c49fd9c070a9d340889a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.757272 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5f8d6960d51e4a63a42f434099952825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.757856 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f8d6960d51e4a63a42f434099952825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.759403 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_30c516ddaa234806bb557849ada87562" } new_table_name: "default.Table_692756cbec294f19a8cd798b13128ca4" modify_external_catalogs: true
I20240503 14:12:16.760362 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 188d7b644d2842109f1e0c124903ddd9 (DEFAULT_TABLE table=default.Table_2ce3b6f2b77b4607856fc2a64ead5bb5 [id=db393a7e2a83466385ffc0af3d2789c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.760732 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188d7b644d2842109f1e0c124903ddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.761744 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_44fe718f4ee1446bba6f7cd746b0154a" } new_table_name: "default.Table_ff5fa02257a5423785ae99a66c759e9c" modify_external_catalogs: true
I20240503 14:12:16.763054 28614 tablet_bootstrap.cc:492] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.763198 28614 ts_tablet_manager.cc:1367] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.063s user 0.000s sys 0.007s
I20240503 14:12:16.763188 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 (DEFAULT_TABLE table=default.Table_1b9a58dce8ab4927bd0bd665004b2e3b [id=0d0ab0d2fe1b4c0ea6e23c4122255dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.763553 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281667db9a9c43d3b9d7dd9ee14b4b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.763885 28614 raft_consensus.cc:348] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.764107 28614 raft_consensus.cc:374] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.764168 28614 raft_consensus.cc:729] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.764150 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3c5bac1089fd4f629005b1381e2bca47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.764757 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c5bac1089fd4f629005b1381e2bca47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.765920 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 5750311061fa4a62a9fed5086fa3e324 (DEFAULT_TABLE table=default.Table_f9806bd6f25a48bfb0a4a65437072534 [id=879354530b66475384a81c299a33d896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.765969 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 885dc7aabc044d10ac149432c8597212 (DEFAULT_TABLE table=default.Table_8796e63f1b454ff08f86549980fc079d [id=63d734176bd644798e3a0e8f69d9b8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.766343 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5750311061fa4a62a9fed5086fa3e324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.752785 29429 leader_election.cc:288] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:16.772984 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 885dc7aabc044d10ac149432c8597212 (DEFAULT_TABLE table=default.Table_8796e63f1b454ff08f86549980fc079d [id=63d734176bd644798e3a0e8f69d9b8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.773404 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885dc7aabc044d10ac149432c8597212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.775627 28327 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f7c221cbc467dabfe5665ccd07d48 (DEFAULT_TABLE table=default.Table_fc189ad8c19041f7814fac6de9f482cb [id=b7f4f49135e245dc975ffd7eb0589a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.776611 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f7c221cbc467dabfe5665ccd07d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.778884 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885dc7aabc044d10ac149432c8597212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.779322 28427 leader_election.cc:302] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:16.779606 29516 raft_consensus.cc:2781] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.779693 29516 raft_consensus.cc:482] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.779759 29516 raft_consensus.cc:3037] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.782426 29516 raft_consensus.cc:504] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.783655 29516 leader_election.cc:288] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:16.786481 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b90e9563b4c4cef8ee7964eaa96e285 consensus state has no leader
I20240503 14:12:16.789152 28604 tablet_bootstrap.cc:492] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:16.789114 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e3c825867cf84b72bfb708fcc39f6acb" } modify_external_catalogs: true
I20240503 14:12:16.789294 28604 ts_tablet_manager.cc:1367] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.225s user 0.003s sys 0.000s
I20240503 14:12:16.789350 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e3c825867cf84b72bfb708fcc39f6acb" } modify_external_catalogs: true
I20240503 14:12:16.790176 28604 raft_consensus.cc:348] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.790403 28604 raft_consensus.cc:374] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.790380 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet b7d4e4a1f0b14406be957426a45bf45d (table default.Table_d3937b734026471d8da300845379e2dd [id=27afd28af9ee45a9ad0aad92898ce78c]) successfully deleted
I20240503 14:12:16.790462 28604 raft_consensus.cc:729] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.790838 28653 tablet_replica.cc:330] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:16.791460 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2aadd33c71aa4953a09c3d7a747e65da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.792200 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aadd33c71aa4953a09c3d7a747e65da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.792284 28604 consensus_queue.cc:260] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.793885 29646 tablet.cc:1722] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:16.794303 28653 raft_consensus.cc:2229] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.797394 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2404807f8500437ababca53f6388b636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.801702 29644 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 14:12:16.802872 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240abf8c2b2343408fc98a8dac1fdf09" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:16.803112 28356 raft_consensus.cc:3037] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.807061 28356 raft_consensus.cc:2454] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:16.807726 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f6061918660e4070a0c8339cb5009ff8
I20240503 14:12:16.811823 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet f6061918660e4070a0c8339cb5009ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:16.812255 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet f6061918660e4070a0c8339cb5009ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:16.817404 29456 consensus_queue.cc:1035] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.817983 28661 ts_tablet_manager.cc:1888] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:16.818127 28661 log.cc:1196] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/1044f34588d045ce898435b61a90503f
I20240503 14:12:16.818509 28661 ts_tablet_manager.cc:1909] T 1044f34588d045ce898435b61a90503f P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:16.818763 28603 ts_tablet_manager.cc:1888] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:12:16.818969 28603 log.cc:1196] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/dc337647c7cd402e91720478380f4696
I20240503 14:12:16.819377 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 547) took 2513 ms. Trace:
I20240503 14:12:16.819468 28661 rpcz_store.cc:276] 0503 14:12:14.306462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.306657 (+ 195us) service_pool.cc:225] Handling call
0503 14:12:16.648759 (+2342102us) ts_tablet_manager.cc:1120] Deleting tablet 1044f34588d045ce898435b61a90503f
0503 14:12:16.648771 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:16.786172 (+137401us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.817859 (+ 31687us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:16.819352 (+ 1493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2338152}
I20240503 14:12:16.819731 28603 ts_tablet_manager.cc:1909] T dc337647c7cd402e91720478380f4696 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.819782 28661 tablet_replica.cc:330] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:16.819964 28661 raft_consensus.cc:2229] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:16.824748 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240abf8c2b2343408fc98a8dac1fdf09" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:16.825027 28212 raft_consensus.cc:3037] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.829236 28653 raft_consensus.cc:2258] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:16.829298 28427 consensus_peers.cc:485] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6dbb59d98834ab6bbac4abfa92681e3. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:16.829466 28427 leader_election.cc:341] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: a7295c2d3bd34e20a69986aadc9c7017
I20240503 14:12:16.829675 28427 leader_election.cc:302] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:16.830312 28653 ts_tablet_manager.cc:1875] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.830924 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fb411e327c414a45b69f242c2113111f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.831640 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb411e327c414a45b69f242c2113111f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:16.837055 28327 tablet_service.cc:1467] Processing CreateTablet for tablet d859044b70fc48e8b62bf6a82ed69d9f (DEFAULT_TABLE table=default.Table_1145b045780c4d1e8fedd333eea661d4 [id=114dfd39b4ea4958a08161a34e4a1f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.837055 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 328c3c9882dd49fb931404e9109bce17 (DEFAULT_TABLE table=default.Table_6e5d5fafa65145abbbad53d9c46f82bf [id=037b69fd14b74295b180bf6f1aa86065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.837520 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328c3c9882dd49fb931404e9109bce17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.840261 29615 raft_consensus.cc:482] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.840443 29615 raft_consensus.cc:504] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.841200 29615 leader_election.cc:288] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.842044 28185 tablet_service.cc:1467] Processing CreateTablet for tablet c29aad1eee3541c5b483b193d769a8da (DEFAULT_TABLE table=default.Table_c64ecffdd4a84ba5bc64aba80e1de1ab [id=cf9c1b19400c41898e376f457d858565]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.842162 28330 tablet_service.cc:1467] Processing CreateTablet for tablet c29aad1eee3541c5b483b193d769a8da (DEFAULT_TABLE table=default.Table_c64ecffdd4a84ba5bc64aba80e1de1ab [id=cf9c1b19400c41898e376f457d858565]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.842468 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29aad1eee3541c5b483b193d769a8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.842590 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df63f75c157a4e928d4dbbda06cfa5ce" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:16.842958 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 328c3c9882dd49fb931404e9109bce17 (DEFAULT_TABLE table=default.Table_6e5d5fafa65145abbbad53d9c46f82bf [id=037b69fd14b74295b180bf6f1aa86065]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:16.843369 28150 leader_election.cc:341] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.846629 28212 raft_consensus.cc:2454] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:16.848393 28614 consensus_queue.cc:260] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.849221 28614 ts_tablet_manager.cc:1398] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.086s user 0.001s sys 0.001s
I20240503 14:12:16.851087 28614 tablet_bootstrap.cc:492] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:16.856274 28179 tablet_service.cc:1467] Processing CreateTablet for tablet d859044b70fc48e8b62bf6a82ed69d9f (DEFAULT_TABLE table=default.Table_1145b045780c4d1e8fedd333eea661d4 [id=114dfd39b4ea4958a08161a34e4a1f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.856885 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d859044b70fc48e8b62bf6a82ed69d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.862495 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df63f75c157a4e928d4dbbda06cfa5ce" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.862859 28600 tablet_bootstrap.cc:492] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.863035 28600 ts_tablet_manager.cc:1367] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.158s user 0.005s sys 0.000s
W20240503 14:12:16.863224 28150 leader_election.cc:341] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.863365 28150 leader_election.cc:302] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.863837 28600 raft_consensus.cc:348] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.864065 28600 raft_consensus.cc:374] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.864136 28600 raft_consensus.cc:729] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.864359 28600 consensus_queue.cc:260] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.864475 28604 ts_tablet_manager.cc:1398] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20240503 14:12:16.864888 28600 ts_tablet_manager.cc:1398] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:16.863848 29615 raft_consensus.cc:2726] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.869812 28026 master_service.cc:946] ReplaceTablet: received request to replace tablet b25c852aa7aa4fbfafdacec43f5bc426 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:16.870808 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_749e46738ab841d4b031fea3fb61e96b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.871575 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_749e46738ab841d4b031fea3fb61e96b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:16.873730 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a703cfbaad2a4a618abb9e07b4930cee consensus state has no leader
I20240503 14:12:16.873849 29516 raft_consensus.cc:2781] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.874001 29516 raft_consensus.cc:686] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:16.874305 29516 consensus_queue.cc:237] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.874347 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0453cf78e30f45ffba602e0ca759e01c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:16.875036 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0453cf78e30f45ffba602e0ca759e01c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:16.876149 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 46486.3us
I20240503 14:12:16.877461 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 5750311061fa4a62a9fed5086fa3e324 (DEFAULT_TABLE table=default.Table_f9806bd6f25a48bfb0a4a65437072534 [id=879354530b66475384a81c299a33d896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.878093 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5750311061fa4a62a9fed5086fa3e324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.888408 29649 tablet.cc:1722] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:16.891742 28600 tablet_bootstrap.cc:492] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.893427 28600 tablet_bootstrap.cc:654] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.894809 29650 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240503 14:12:16.901728 28264 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:16.904826 29645 raft_consensus.cc:482] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.905090 29645 raft_consensus.cc:504] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.905808 29645 leader_election.cc:288] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:16.906008 28604 tablet_bootstrap.cc:492] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.908311 29602 raft_consensus.cc:482] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.908486 29602 raft_consensus.cc:504] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.908540 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 885dc7aabc044d10ac149432c8597212 (DEFAULT_TABLE table=default.Table_8796e63f1b454ff08f86549980fc079d [id=63d734176bd644798e3a0e8f69d9b8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.909021 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885dc7aabc044d10ac149432c8597212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.909233 29602 leader_election.cc:288] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:16.909767 28604 tablet_bootstrap.cc:654] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.910717 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2404807f8500437ababca53f6388b636" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:16.911825 28150 leader_election.cc:341] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.916113 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet f6061918660e4070a0c8339cb5009ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:16.916636 28469 tablet_service.cc:1467] Processing CreateTablet for tablet d859044b70fc48e8b62bf6a82ed69d9f (DEFAULT_TABLE table=default.Table_1145b045780c4d1e8fedd333eea661d4 [id=114dfd39b4ea4958a08161a34e4a1f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.917096 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d859044b70fc48e8b62bf6a82ed69d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.917064 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 1044f34588d045ce898435b61a90503f (table default.Table_6673b8d0a18d42c3a18f6e611c596580 [id=3c43a0fcf51143179a1ed0e1b8e333c1]) successfully deleted
I20240503 14:12:16.920684 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 328c3c9882dd49fb931404e9109bce17 (DEFAULT_TABLE table=default.Table_6e5d5fafa65145abbbad53d9c46f82bf [id=037b69fd14b74295b180bf6f1aa86065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.921180 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328c3c9882dd49fb931404e9109bce17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.924595 28458 tablet_service.cc:1467] Processing CreateTablet for tablet c29aad1eee3541c5b483b193d769a8da (DEFAULT_TABLE table=default.Table_c64ecffdd4a84ba5bc64aba80e1de1ab [id=cf9c1b19400c41898e376f457d858565]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.924937 28604 tablet_bootstrap.cc:492] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:16.925038 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29aad1eee3541c5b483b193d769a8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.925146 28604 ts_tablet_manager.cc:1367] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240503 14:12:16.925922 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29aad1eee3541c5b483b193d769a8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.926404 28604 raft_consensus.cc:348] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.926805 28604 raft_consensus.cc:374] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.926893 28604 raft_consensus.cc:729] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.927320 28604 consensus_queue.cc:260] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.928725 28604 ts_tablet_manager.cc:1398] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.929837 28604 tablet_bootstrap.cc:492] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:16.930220 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d859044b70fc48e8b62bf6a82ed69d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.931155 28614 tablet_bootstrap.cc:654] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.935817 28604 tablet_bootstrap.cc:654] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.938385 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2404807f8500437ababca53f6388b636" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.939808 28653 ts_tablet_manager.cc:1888] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:12:16.940162 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ba571a81eb4240a4a4a4a08cb879c5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:16.940435 28353 raft_consensus.cc:2379] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:16.954809 29456 consensus_queue.cc:1035] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.956955 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328c3c9882dd49fb931404e9109bce17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.968628 28603 tablet_replica.cc:330] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:16.968871 28603 raft_consensus.cc:2229] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:16.969090 28603 raft_consensus.cc:2258] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.969203 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet dc337647c7cd402e91720478380f4696 (table default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd]) successfully deleted
I20240503 14:12:16.969748 28603 ts_tablet_manager.cc:1875] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:16.973526 29656 tablet.cc:1722] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:16.977116 28661 pending_rounds.cc:62] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:16.977217 28661 pending_rounds.cc:68] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597712514801664 op_type: NO_OP noop_request { }
I20240503 14:12:16.977375 28661 raft_consensus.cc:2866] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:16.977469 28661 raft_consensus.cc:2258] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:16.978063 28661 ts_tablet_manager.cc:1875] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:16.980353 28427 leader_election.cc:341] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a7295c2d3bd34e20a69986aadc9c7017
I20240503 14:12:16.980506 28427 leader_election.cc:302] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:16.980964 28427 consensus_peers.cc:485] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed9262a7e828489ab51a745068c737df. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:16.976266 28603 ts_tablet_manager.cc:1888] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:16.981477 28603 log.cc:1196] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/47ba571a81eb4240a4a4a4a08cb879c5
I20240503 14:12:16.981945 28603 ts_tablet_manager.cc:1909] T 47ba571a81eb4240a4a4a4a08cb879c5 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:16.977983 29657 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20240503 14:12:16.940946 28150 leader_election.cc:341] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:16.982854 28150 leader_election.cc:302] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:16.983969 29602 raft_consensus.cc:2726] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:16.956727 28600 tablet_bootstrap.cc:492] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.984604 28600 ts_tablet_manager.cc:1367] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.003s
I20240503 14:12:16.985512 28600 raft_consensus.cc:348] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.985881 28600 raft_consensus.cc:374] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.986003 28600 raft_consensus.cc:729] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.986354 28600 consensus_queue.cc:260] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.987129 28600 ts_tablet_manager.cc:1398] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:16.987527 28600 tablet_bootstrap.cc:492] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:16.989121 28600 tablet_bootstrap.cc:654] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.991314 28653 log.cc:1196] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/dc337647c7cd402e91720478380f4696
I20240503 14:12:16.992563 28600 tablet_bootstrap.cc:492] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:16.994407 28600 ts_tablet_manager.cc:1367] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:12:16.994091 28653 ts_tablet_manager.cc:1909] T dc337647c7cd402e91720478380f4696 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:16.996311 28600 raft_consensus.cc:348] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:16.997280 28600 raft_consensus.cc:374] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.997637 28600 raft_consensus.cc:729] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.998039 28614 tablet_bootstrap.cc:492] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:16.998783 28600 consensus_queue.cc:260] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.000263 28614 ts_tablet_manager.cc:1367] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.149s user 0.004s sys 0.001s
I20240503 14:12:17.001307 28600 ts_tablet_manager.cc:1398] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 14:12:17.001715 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet dc337647c7cd402e91720478380f4696 (table default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd]) successfully deleted
W20240503 14:12:17.002164 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0c7779f7a9544e17aea7135eab2ccd5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c7779f7a9544e17aea7135eab2ccd5f already in progress: creating tablet
I20240503 14:12:17.002215 28653 tablet_replica.cc:330] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:17.002537 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2b52ef7ecedf4273939406581d843980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b52ef7ecedf4273939406581d843980 already in progress: creating tablet
I20240503 14:12:17.002589 28600 tablet_bootstrap.cc:492] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:17.002882 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet af0137d7b1034a79bf4947c7d3f3fd57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af0137d7b1034a79bf4947c7d3f3fd57 already in progress: creating tablet
W20240503 14:12:17.003237 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 64f226db2fdb47ba9472a3aa65b953c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f226db2fdb47ba9472a3aa65b953c3 already in progress: creating tablet
I20240503 14:12:17.003544 29516 raft_consensus.cc:2726] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.004313 29652 raft_consensus.cc:482] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.004541 29652 raft_consensus.cc:504] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.004745 28604 tablet_bootstrap.cc:492] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.003180 28653 raft_consensus.cc:2229] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.005043 28604 ts_tablet_manager.cc:1367] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.075s user 0.017s sys 0.012s
I20240503 14:12:17.005542 28653 raft_consensus.cc:2258] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.006052 28604 raft_consensus.cc:348] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.006270 28604 raft_consensus.cc:374] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.006331 28604 raft_consensus.cc:729] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.006562 28604 consensus_queue.cc:260] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.007232 28604 ts_tablet_manager.cc:1398] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:17.008070 29652 leader_election.cc:288] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.008327 28653 ts_tablet_manager.cc:1875] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.008765 28604 tablet_bootstrap.cc:492] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.010507 28604 tablet_bootstrap.cc:654] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.012475 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0926e523ef454a2089f4d40a328c1c6d
W20240503 14:12:17.012699 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9c7b5fb9b4a483c8292f68387591034 consensus state has no leader
I20240503 14:12:17.013392 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_871098e8af8c4007bb6ec13983c587de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:17.014015 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7b4a078c66b0477180bed5b7b139de58" } new_table_name: "default.Table_960d64d809814618909c0ab5c4602156" modify_external_catalogs: true
W20240503 14:12:17.014199 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_871098e8af8c4007bb6ec13983c587de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.015777 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7218cefb13ea4983b2111ed5fd0c997b
I20240503 14:12:17.016813 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d9142b8d36e542648c29cf7dcce32d4c" } modify_external_catalogs: true
I20240503 14:12:17.016942 28030 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8] alter complete (version 2)
I20240503 14:12:17.017127 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d9142b8d36e542648c29cf7dcce32d4c" } modify_external_catalogs: true
I20240503 14:12:17.017343 28614 raft_consensus.cc:348] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.017580 28614 raft_consensus.cc:374] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.017637 28614 raft_consensus.cc:729] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.017876 28614 consensus_queue.cc:260] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.017985 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9150e78ff2d94d1dba8f6d1fd1727db2" } modify_external_catalogs: true
I20240503 14:12:17.018220 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9150e78ff2d94d1dba8f6d1fd1727db2" } modify_external_catalogs: true
I20240503 14:12:17.020036 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 007c84fd9d9247728b0babd48689aba8 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:17.020288 28427 consensus_peers.cc:485] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6dbb59d98834ab6bbac4abfa92681e3. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:17.020503 28427 leader_election.cc:341] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 47ba571a81eb4240a4a4a4a08cb879c5
I20240503 14:12:17.020466 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9c932c51a0a844c38b521b84c50940a8" } new_table_name: "default.Table_b7b3cecd524e40feb9c6251d0403b764" modify_external_catalogs: true
I20240503 14:12:17.022469 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_091731e3847544b1b411f7fe14d3182a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.023198 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_091731e3847544b1b411f7fe14d3182a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.035907 29456 raft_consensus.cc:482] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.036140 29456 raft_consensus.cc:504] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.036808 28614 ts_tablet_manager.cc:1398] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240503 14:12:17.036919 29456 leader_election.cc:288] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.037048 29456 raft_consensus.cc:482] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.037153 29456 raft_consensus.cc:504] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.037803 29456 leader_election.cc:288] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.038110 28427 leader_election.cc:302] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.038414 29516 raft_consensus.cc:3037] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.013393 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_986bd23778564daa8b25045aa3d81420"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.040172 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_986bd23778564daa8b25045aa3d81420 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.041184 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135b60660f448a7974520f2f87cf93a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.041357 29516 raft_consensus.cc:2726] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.042342 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fb79fcf30d485798c64c815e372dc1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:17.042366 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ace9146e755549ba882df639de92097f" } new_table_name: "default.Table_5dcdff74f1f7407084a45dc9fe588103" modify_external_catalogs: true
I20240503 14:12:17.042641 28214 raft_consensus.cc:2454] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:17.046897 28290 leader_election.cc:302] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:17.047343 29595 raft_consensus.cc:2781] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.047426 29595 raft_consensus.cc:482] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.047482 29595 raft_consensus.cc:3037] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:17.049113 28427 leader_election.cc:341] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: e135b60660f448a7974520f2f87cf93a
I20240503 14:12:17.050693 29595 raft_consensus.cc:504] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.051414 29595 leader_election.cc:288] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:17.052038 28290 leader_election.cc:341] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 1603e2360041427f8635782e812b6431
I20240503 14:12:17.054883 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fb79fcf30d485798c64c815e372dc1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:17.055131 28214 raft_consensus.cc:3037] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.058730 28214 raft_consensus.cc:2454] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:17.059906 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1603e2360041427f8635782e812b6431" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.060648 29652 raft_consensus.cc:482] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.060683 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fb79fcf30d485798c64c815e372dc1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.061230 29652 raft_consensus.cc:504] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:17.061544 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 53239.9us
I20240503 14:12:17.061945 29652 leader_election.cc:288] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.063024 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 7218cefb13ea4983b2111ed5fd0c997b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.063814 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cdd090bf0f493f8b92341369378532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.069383 28600 tablet_bootstrap.cc:654] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:17.073695 28427 consensus_peers.cc:485] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed9262a7e828489ab51a745068c737df. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:17.075747 28427 leader_election.cc:302] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.080230 28604 tablet_bootstrap.cc:492] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.080668 28604 ts_tablet_manager.cc:1367] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.002s
I20240503 14:12:17.083765 29652 raft_consensus.cc:2726] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.084750 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fb79fcf30d485798c64c815e372dc1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:17.084939 28604 raft_consensus.cc:348] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.085354 28604 raft_consensus.cc:374] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.085548 28604 raft_consensus.cc:729] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.085963 28604 consensus_queue.cc:260] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.087420 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0926e523ef454a2089f4d40a328c1c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:17.087563 28290 leader_election.cc:341] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.087603 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 7218cefb13ea4983b2111ed5fd0c997b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.087911 28290 leader_election.cc:302] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.088423 29595 raft_consensus.cc:2726] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.088907 28600 tablet_bootstrap.cc:492] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.089355 28600 ts_tablet_manager.cc:1367] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.005s
I20240503 14:12:17.089442 29602 raft_consensus.cc:482] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.089656 29602 raft_consensus.cc:504] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.090474 29602 leader_election.cc:288] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:17.092516 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5bf863dabf6647618732a03af84efab3 consensus state has no leader
I20240503 14:12:17.094851 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ffcfc89a415e4c68b6a54318755fe0d7
I20240503 14:12:17.096511 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf5d153ab5142c0902cc6e96629f412" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:17.098009 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6c90869b5034128864faaf05697c230 consensus state has no leader
I20240503 14:12:17.098130 28603 tablet_replica.cc:330] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:17.098337 28603 raft_consensus.cc:2229] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:17.098471 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 788e4bfa1e414630939bc19da1ede3fb consensus state has no leader
I20240503 14:12:17.098593 28603 pending_rounds.cc:62] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:17.098637 28603 pending_rounds.cc:68] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597716395073536 op_type: NO_OP noop_request { }
I20240503 14:12:17.098754 28603 raft_consensus.cc:2866] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:17.098846 28603 raft_consensus.cc:2258] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.098819 28604 ts_tablet_manager.cc:1398] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 14:12:17.099475 28604 tablet_bootstrap.cc:492] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.099752 28603 ts_tablet_manager.cc:1875] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:17.101284 28150 leader_election.cc:341] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.102423 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf5d153ab5142c0902cc6e96629f412" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:17.103428 28150 leader_election.cc:341] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.103564 28150 leader_election.cc:302] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:17.103790 28290 leader_election.cc:341] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.103783 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:17.104033 28214 raft_consensus.cc:2379] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
W20240503 14:12:17.104373 28290 leader_election.cc:341] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.104707 29602 raft_consensus.cc:2726] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.104979 28290 leader_election.cc:302] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.105429 29595 raft_consensus.cc:2781] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.105960 29595 raft_consensus.cc:686] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:17.106213 28600 raft_consensus.cc:348] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.106339 29595 consensus_queue.cc:237] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.106657 28600 raft_consensus.cc:374] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.106806 28600 raft_consensus.cc:729] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.107249 28600 consensus_queue.cc:260] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.108675 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcfc89a415e4c68b6a54318755fe0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.110612 28614 tablet_bootstrap.cc:492] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:17.110743 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ce1f05b05bd3429eacedb14a5f34b7da" } modify_external_catalogs: true
I20240503 14:12:17.111045 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ce1f05b05bd3429eacedb14a5f34b7da" } modify_external_catalogs: true
I20240503 14:12:17.117765 29602 raft_consensus.cc:482] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.117954 29602 raft_consensus.cc:504] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.118695 29602 leader_election.cc:288] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.121803 28614 tablet_bootstrap.cc:654] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.122608 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7218cefb13ea4983b2111ed5fd0c997b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:17.124296 28150 leader_election.cc:341] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.124433 28603 ts_tablet_manager.cc:1888] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:17.124557 28603 log.cc:1196] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e135b60660f448a7974520f2f87cf93a
I20240503 14:12:17.124915 28603 ts_tablet_manager.cc:1909] T e135b60660f448a7974520f2f87cf93a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:17.125412 28603 tablet_replica.cc:330] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:17.125622 28603 raft_consensus.cc:2229] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.125824 28603 raft_consensus.cc:2258] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.125852 28600 ts_tablet_manager.cc:1398] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 14:12:17.126421 28603 ts_tablet_manager.cc:1875] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.115727 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 0926e523ef454a2089f4d40a328c1c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.135402 28600 tablet_bootstrap.cc:492] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.138916 28653 ts_tablet_manager.cc:1888] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.140035 28653 log.cc:1196] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/95d9985eecda4f3194c3251216aa58ee
I20240503 14:12:17.142016 28653 ts_tablet_manager.cc:1909] T 95d9985eecda4f3194c3251216aa58ee P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:17.144572 29602 raft_consensus.cc:482] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.144732 29602 raft_consensus.cc:504] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.145388 29602 leader_election.cc:288] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.148224 29602 raft_consensus.cc:482] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.148367 29602 raft_consensus.cc:504] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.148973 29602 leader_election.cc:288] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.150100 28600 tablet_bootstrap.cc:654] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.154314 28653 tablet_replica.cc:330] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:17.155128 28653 raft_consensus.cc:2229] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.155743 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 47ba571a81eb4240a4a4a4a08cb879c5 (table default.Table_852b77a0ef2e4ee6bda7be721348aff8 [id=0a16a4e85e1842f1a2880d2873d363b1]) successfully deleted
I20240503 14:12:17.157444 28653 raft_consensus.cc:2258] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.158888 28653 ts_tablet_manager.cc:1875] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.164266 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d121667544004d9e83e4daf3f58f7473" } new_table_name: "default.Table_cdaf5876f13c4bd3adb7cb8aea0cfc88" modify_external_catalogs: true
W20240503 14:12:17.165437 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 0a6a58db50f849a8be729c10dbe03e76 (table default.Table_3bf21cdd53304e70a9c5b910bc33ce19 [id=6199baccf6a9435f8d53d80f58531766], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a6a58db50f849a8be729c10dbe03e76 consensus state has no leader
I20240503 14:12:17.115253 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 0926e523ef454a2089f4d40a328c1c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.116828 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 7218cefb13ea4983b2111ed5fd0c997b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.119980 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7218cefb13ea4983b2111ed5fd0c997b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.169067 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 95d9985eecda4f3194c3251216aa58ee (table default.Table_627b41e76476426da74d6722301bc59f [id=97e7cb47edc44a5e9041be7ad4bc6f47]) successfully deleted
I20240503 14:12:17.146322 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60206e8e73e84e409ed6aba222d9ff4b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:17.169477 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f080c95ff5ca41e0b3d27382040107e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f080c95ff5ca41e0b3d27382040107e9 already in progress: creating tablet
I20240503 14:12:17.156374 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ba571a81eb4240a4a4a4a08cb879c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.149886 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2331aec871824a238f3f64de217d7092" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.170009 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b9af7d02fc5a42a7a09ecbfb33775ccf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:17.156508 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcfc89a415e4c68b6a54318755fe0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:17.171432 28150 leader_election.cc:341] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.171587 28150 leader_election.cc:302] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:17.172029 28150 leader_election.cc:341] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.172247 28150 leader_election.cc:341] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.172441 29602 raft_consensus.cc:2726] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.117409 28661 ts_tablet_manager.cc:1888] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:17.173112 28661 log.cc:1196] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0b139ab0b5864024ac48215d46b0a7bd
I20240503 14:12:17.173307 28600 tablet_bootstrap.cc:492] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.173602 28600 ts_tablet_manager.cc:1367] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20240503 14:12:17.174685 28661 ts_tablet_manager.cc:1909] T 0b139ab0b5864024ac48215d46b0a7bd P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.175102 28600 raft_consensus.cc:348] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.102923 28604 tablet_bootstrap.cc:654] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.175616 28600 raft_consensus.cc:374] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.175839 28600 raft_consensus.cc:729] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.176425 28600 consensus_queue.cc:260] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.134454 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcfc89a415e4c68b6a54318755fe0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.134495 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0926e523ef454a2089f4d40a328c1c6d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.173580 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e135b60660f448a7974520f2f87cf93a (table default.Table_00e404138f154498b730db1de48ded05 [id=6edae5b6460546ae80b1f25ffec68d48]) successfully deleted
I20240503 14:12:17.179309 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 06fb79fcf30d485798c64c815e372dc1 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:17.146638 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60206e8e73e84e409ed6aba222d9ff4b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.150913 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2331aec871824a238f3f64de217d7092" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.178499 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_39c390a1cebb4679b78b0e36750364d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.181789 28150 leader_election.cc:341] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.181941 28150 leader_election.cc:302] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:17.182253 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39c390a1cebb4679b78b0e36750364d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:17.183045 28150 leader_election.cc:341] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.183176 28150 leader_election.cc:302] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:17.185487 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9af7d02fc5a42a7a09ecbfb33775ccf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.186864 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a32b6995818041778617a0d6bf936c8c
I20240503 14:12:17.187924 29615 raft_consensus.cc:2726] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.188174 29615 raft_consensus.cc:2726] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.194339 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet a32b6995818041778617a0d6bf936c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.194371 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet a32b6995818041778617a0d6bf936c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.199921 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 548) took 2857 ms. Trace:
I20240503 14:12:17.200189 28661 rpcz_store.cc:276] 0503 14:12:14.342055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.342131 (+ 76us) service_pool.cc:225] Handling call
0503 14:12:16.819727 (+2477596us) ts_tablet_manager.cc:1120] Deleting tablet 0b139ab0b5864024ac48215d46b0a7bd
0503 14:12:16.819737 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.046090 (+226353us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.115852 (+ 69762us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.199884 (+ 84032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477322}
I20240503 14:12:17.200855 28661 tablet_replica.cc:330] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.201543 28600 ts_tablet_manager.cc:1398] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20240503 14:12:17.205651 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0b139ab0b5864024ac48215d46b0a7bd (table default.Table_52932a5c26744566bfc744e2ed1d8aee [id=5c194c0fc1c94da0a40f0f3f0d7484b7]) successfully deleted
I20240503 14:12:17.215144 28600 tablet_bootstrap.cc:492] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.215662 28614 tablet_bootstrap.cc:492] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:17.204602 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:17.205344 28427 leader_election.cc:302] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.208299 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet a32b6995818041778617a0d6bf936c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.216420 29595 consensus_queue.cc:1035] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:17.224433 28614 ts_tablet_manager.cc:1367] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.005s
I20240503 14:12:17.225214 28614 raft_consensus.cc:348] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.225523 28614 raft_consensus.cc:374] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.225591 28614 raft_consensus.cc:729] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.220563 29516 raft_consensus.cc:3037] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.226868 28603 ts_tablet_manager.cc:1888] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.228662 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7c688a2e434147e582bcf88eeafc227b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.229938 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c688a2e434147e582bcf88eeafc227b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.230191 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:17.231665 29516 raft_consensus.cc:2726] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.220067 28661 raft_consensus.cc:2229] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:17.231968 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.214625 28604 tablet_bootstrap.cc:492] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.232265 28604 ts_tablet_manager.cc:1367] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.001s
I20240503 14:12:17.233330 28604 raft_consensus.cc:348] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.233547 28604 raft_consensus.cc:374] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.233603 28604 raft_consensus.cc:729] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.233899 28604 consensus_queue.cc:260] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.234692 28604 ts_tablet_manager.cc:1398] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:17.235169 28604 tablet_bootstrap.cc:492] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.235227 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ec3edf11cc5746799ddb3623736b23fe" } new_table_name: "default.Table_701c15f6af2d4365aa030b14bbd1d82a" modify_external_catalogs: true
I20240503 14:12:17.236932 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_98e9ee5715624c0c804e368311ceabb5" } modify_external_catalogs: true
I20240503 14:12:17.238492 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_98e9ee5715624c0c804e368311ceabb5" } modify_external_catalogs: true
W20240503 14:12:17.241765 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cd037e135904e2cab09cbf39fed0b13 consensus state has no leader
I20240503 14:12:17.242486 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f8c8d1a54b104cbb88a7030052a89be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.243155 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8c8d1a54b104cbb88a7030052a89be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.245090 28603 log.cc:1196] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:17.246639 28603 ts_tablet_manager.cc:1909] T 5775006e9f8f4612aa70dfb316de9ea1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:17.248104 29602 raft_consensus.cc:482] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.248312 29602 raft_consensus.cc:504] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.249102 29602 leader_election.cc:288] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.250674 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140de4d395d54b7696062ed96b8aa89c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:17.251541 28150 leader_election.cc:341] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.250298 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140de4d395d54b7696062ed96b8aa89c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.256217 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc8ca3c970a949fab24298af4538c323" } modify_external_catalogs: true
W20240503 14:12:17.256290 28150 leader_election.cc:341] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.256440 28150 leader_election.cc:302] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.256528 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc8ca3c970a949fab24298af4538c323" } modify_external_catalogs: true
I20240503 14:12:17.256928 29602 raft_consensus.cc:2726] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.257678 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e4311f501e744a4a859e4eef2878127c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.258436 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4311f501e744a4a859e4eef2878127c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:17.258643 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9aaf7d4927674eedbfd486df6a347f7c consensus state has no leader
I20240503 14:12:17.267679 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet b7d4e4a1f0b14406be957426a45bf45d from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:17.270476 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 5775006e9f8f4612aa70dfb316de9ea1 (table default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842]) successfully deleted
I20240503 14:12:17.272480 28661 raft_consensus.cc:2258] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.273247 28661 ts_tablet_manager.cc:1875] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.272269 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:17.276472 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:17.276818 29516 raft_consensus.cc:482] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.276847 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:17.277797 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 5757e6e8703640519f7d38754cc0c8ce from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:17.276983 29516 raft_consensus.cc:504] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.278730 29516 leader_election.cc:288] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.279132 28026 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:17.280965 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:17.281366 28030 catalog_manager.cc:5484] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:17.281905 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:17.281910 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1603e2360041427f8635782e812b6431" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.282162 28352 raft_consensus.cc:2454] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:17.282550 28614 consensus_queue.cc:260] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.285159 28614 ts_tablet_manager.cc:1398] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.061s user 0.001s sys 0.001s
I20240503 14:12:17.285631 28614 tablet_bootstrap.cc:492] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:17.287809 28614 tablet_bootstrap.cc:654] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:17.288305 28427 leader_election.cc:341] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 1603e2360041427f8635782e812b6431
I20240503 14:12:17.289821 28600 tablet_bootstrap.cc:654] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.289974 28653 ts_tablet_manager.cc:1888] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.291532 28653 log.cc:1196] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/06fb79fcf30d485798c64c815e372dc1
W20240503 14:12:17.295969 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0c7779f7a9544e17aea7135eab2ccd5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c7779f7a9544e17aea7135eab2ccd5f already in progress: creating tablet
W20240503 14:12:17.296684 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2b52ef7ecedf4273939406581d843980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b52ef7ecedf4273939406581d843980 already in progress: creating tablet
I20240503 14:12:17.296648 28427 leader_election.cc:302] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:17.296983 29516 raft_consensus.cc:2781] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.297124 29516 raft_consensus.cc:482] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.297258 29516 raft_consensus.cc:3037] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.296849 28653 ts_tablet_manager.cc:1909] T 06fb79fcf30d485798c64c815e372dc1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:17.298520 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4631565e9a432689419608fdd70249 (DEFAULT_TABLE table=default.Table_85a010a99b4d4402a290c308cf51e85c [id=8b8a0dab39b74444bd534e1a0a8aefca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.299036 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4631565e9a432689419608fdd70249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.301045 28603 tablet_replica.cc:330] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:17.301275 28603 raft_consensus.cc:2229] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:17.301523 28603 pending_rounds.cc:62] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:17.301561 28603 pending_rounds.cc:68] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597719991410688 op_type: NO_OP noop_request { }
I20240503 14:12:17.301725 28603 raft_consensus.cc:2866] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:17.301827 28603 raft_consensus.cc:2258] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.301975 29516 raft_consensus.cc:504] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.302654 29516 leader_election.cc:288] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.303835 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1603e2360041427f8635782e812b6431" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:17.304101 28352 raft_consensus.cc:3037] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.304894 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 6874d970e4354070b11060c6b861d7bf (DEFAULT_TABLE table=default.Table_7cafc45f1ec54604b2608f91acef4847 [id=9ec88f807cfa456485eba5ed43c30dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.304982 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4631565e9a432689419608fdd70249 (DEFAULT_TABLE table=default.Table_85a010a99b4d4402a290c308cf51e85c [id=8b8a0dab39b74444bd534e1a0a8aefca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.305337 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6874d970e4354070b11060c6b861d7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.305429 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 6874d970e4354070b11060c6b861d7bf (DEFAULT_TABLE table=default.Table_7cafc45f1ec54604b2608f91acef4847 [id=9ec88f807cfa456485eba5ed43c30dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.304991 28187 tablet_service.cc:1467] Processing CreateTablet for tablet b0c72d1569de4810853eebb533d77c9b (DEFAULT_TABLE table=default.Table_425624bfb64a4882893b569d3044c2c0 [id=83ea0916975a4da1a87f345b044688a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.305918 28604 tablet_bootstrap.cc:654] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.306550 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4631565e9a432689419608fdd70249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.306553 28327 tablet_service.cc:1467] Processing CreateTablet for tablet b0c72d1569de4810853eebb533d77c9b (DEFAULT_TABLE table=default.Table_425624bfb64a4882893b569d3044c2c0 [id=83ea0916975a4da1a87f345b044688a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.307580 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4631565e9a432689419608fdd70249 (DEFAULT_TABLE table=default.Table_85a010a99b4d4402a290c308cf51e85c [id=8b8a0dab39b74444bd534e1a0a8aefca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.307747 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 6874d970e4354070b11060c6b861d7bf (DEFAULT_TABLE table=default.Table_7cafc45f1ec54604b2608f91acef4847 [id=9ec88f807cfa456485eba5ed43c30dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.308001 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4631565e9a432689419608fdd70249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.308238 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6874d970e4354070b11060c6b861d7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.311630 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c72d1569de4810853eebb533d77c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.312386 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c72d1569de4810853eebb533d77c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.316385 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6874d970e4354070b11060c6b861d7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.318521 28352 raft_consensus.cc:2454] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:17.319553 28333 tablet_service.cc:1467] Processing CreateTablet for tablet f18ca6c47daa45d9a2623fdb6f6ea53f (DEFAULT_TABLE table=default.Table_691ea23c5adc4b43935e43b5f3749a49 [id=d4dd689795c74bdcacd86023dd677da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.320248 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18ca6c47daa45d9a2623fdb6f6ea53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.320523 28427 leader_election.cc:302] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:17.320732 28472 tablet_service.cc:1467] Processing CreateTablet for tablet b0c72d1569de4810853eebb533d77c9b (DEFAULT_TABLE table=default.Table_425624bfb64a4882893b569d3044c2c0 [id=83ea0916975a4da1a87f345b044688a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.320964 29516 raft_consensus.cc:2781] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.321280 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c72d1569de4810853eebb533d77c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.323078 29516 raft_consensus.cc:686] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:17.323614 29516 consensus_queue.cc:237] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.325968 28603 ts_tablet_manager.cc:1875] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:17.328295 28427 leader_election.cc:341] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 1603e2360041427f8635782e812b6431
I20240503 14:12:17.331632 28194 tablet_service.cc:1467] Processing CreateTablet for tablet f18ca6c47daa45d9a2623fdb6f6ea53f (DEFAULT_TABLE table=default.Table_691ea23c5adc4b43935e43b5f3749a49 [id=d4dd689795c74bdcacd86023dd677da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.331748 28327 tablet_service.cc:1467] Processing CreateTablet for tablet e06bdac2db0b4255be02d561c6fcbaac (DEFAULT_TABLE table=default.Table_ab8c3cf894c9424f9aa6acde9e86ff7f [id=6daa9dd1ca584d1d92d7e15298f3b66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.332224 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06bdac2db0b4255be02d561c6fcbaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.332552 28178 tablet_service.cc:1467] Processing CreateTablet for tablet e06bdac2db0b4255be02d561c6fcbaac (DEFAULT_TABLE table=default.Table_ab8c3cf894c9424f9aa6acde9e86ff7f [id=6daa9dd1ca584d1d92d7e15298f3b66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.334087 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18ca6c47daa45d9a2623fdb6f6ea53f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:17.334699 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet af0137d7b1034a79bf4947c7d3f3fd57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af0137d7b1034a79bf4947c7d3f3fd57 already in progress: creating tablet
W20240503 14:12:17.335081 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 64f226db2fdb47ba9472a3aa65b953c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f226db2fdb47ba9472a3aa65b953c3 already in progress: creating tablet
I20240503 14:12:17.337255 28472 tablet_service.cc:1467] Processing CreateTablet for tablet f18ca6c47daa45d9a2623fdb6f6ea53f (DEFAULT_TABLE table=default.Table_691ea23c5adc4b43935e43b5f3749a49 [id=d4dd689795c74bdcacd86023dd677da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.337740 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18ca6c47daa45d9a2623fdb6f6ea53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.343379 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_501a1223ca764fdb8381a08a543f6bd7" } new_table_name: "default.Table_7dc512798f934fd4a2d847c3f9ae1c16" modify_external_catalogs: true
I20240503 14:12:17.346681 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dffe0d8805804b6a8c235ba8f8772152" } modify_external_catalogs: true
I20240503 14:12:17.346998 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dffe0d8805804b6a8c235ba8f8772152" } modify_external_catalogs: true
I20240503 14:12:17.348771 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06bdac2db0b4255be02d561c6fcbaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.350661 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 4904b0abf5d644a39079b0667e83aaa0 (DEFAULT_TABLE table=default.Table_c9fd8692f46d44adb6c10e08bbccdfa6 [id=c31d2424205a47378579fa5635ce24a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.351506 28604 tablet_bootstrap.cc:492] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.351645 28604 ts_tablet_manager.cc:1367] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.117s user 0.002s sys 0.002s
I20240503 14:12:17.352408 29602 raft_consensus.cc:482] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.352463 28661 ts_tablet_manager.cc:1888] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:17.352576 28661 log.cc:1196] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/8c7d57fc4486457e9848daad79f04356
I20240503 14:12:17.352578 29602 raft_consensus.cc:504] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.353341 29602 leader_election.cc:288] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.354111 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abcb8658fc34c939aa74cf0bd981ae4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.354343 28458 tablet_service.cc:1467] Processing CreateTablet for tablet e06bdac2db0b4255be02d561c6fcbaac (DEFAULT_TABLE table=default.Table_ab8c3cf894c9424f9aa6acde9e86ff7f [id=6daa9dd1ca584d1d92d7e15298f3b66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.354830 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06bdac2db0b4255be02d561c6fcbaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.354838 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abcb8658fc34c939aa74cf0bd981ae4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:17.356199 28150 leader_election.cc:341] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.356411 28150 leader_election.cc:341] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.356482 28150 leader_election.cc:302] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.356595 28604 raft_consensus.cc:348] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.356835 28604 raft_consensus.cc:374] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.356895 28604 raft_consensus.cc:729] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.357291 29602 raft_consensus.cc:2726] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.358462 28604 consensus_queue.cc:260] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.359704 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 4904b0abf5d644a39079b0667e83aaa0 (DEFAULT_TABLE table=default.Table_c9fd8692f46d44adb6c10e08bbccdfa6 [id=c31d2424205a47378579fa5635ce24a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.359687 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 4904b0abf5d644a39079b0667e83aaa0 (DEFAULT_TABLE table=default.Table_c9fd8692f46d44adb6c10e08bbccdfa6 [id=c31d2424205a47378579fa5635ce24a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.359871 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4904b0abf5d644a39079b0667e83aaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.359966 28604 ts_tablet_manager.cc:1398] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:12:17.360148 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4904b0abf5d644a39079b0667e83aaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.360164 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4904b0abf5d644a39079b0667e83aaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.364159 28604 tablet_bootstrap.cc:492] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.364692 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 007c84fd9d9247728b0babd48689aba8
I20240503 14:12:17.364897 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 16fed718124a4640967ab13a1ac2b379
I20240503 14:12:17.365346 28027 catalog_manager.cc:6554] ReplaceTablet: tablet 007c84fd9d9247728b0babd48689aba8 deleted and replaced by tablet 4451a8c3ab1e4fccac7100ac74bb990f
I20240503 14:12:17.364696 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 11d9f47543524fc5921c99aad869035f
I20240503 14:12:17.365888 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f2e867e587ff49c293df696d8dd39cf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.366654 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2e867e587ff49c293df696d8dd39cf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.366871 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9bcc2ed2ad6745709f5987491f698eee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:17.367434 28604 tablet_bootstrap.cc:654] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:17.367499 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bcc2ed2ad6745709f5987491f698eee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.369019 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e21a5dafa82f4660927d96c779a23478
I20240503 14:12:17.369858 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fc86c8af96ac4a70906e1a764f52c59b" } new_table_name: "default.Table_c1e38c13e56749f990b2f0d9c5863be5" modify_external_catalogs: true
I20240503 14:12:17.365882 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_12c636b581474b7998548e4a7d83e95c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.372505 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12c636b581474b7998548e4a7d83e95c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.375362 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 8900cba02fdb46a9bb08943812d3709d (DEFAULT_TABLE table=default.Table_8fc00adb6ca146859c462157a163a3c9 [id=65ace760cd924ba08b7318175705182e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.375852 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8900cba02fdb46a9bb08943812d3709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.379876 28661 ts_tablet_manager.cc:1909] T 8c7d57fc4486457e9848daad79f04356 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.379904 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 8900cba02fdb46a9bb08943812d3709d (DEFAULT_TABLE table=default.Table_8fc00adb6ca146859c462157a163a3c9 [id=65ace760cd924ba08b7318175705182e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.380380 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8900cba02fdb46a9bb08943812d3709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.381271 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 562) took 2922 ms. Trace:
I20240503 14:12:17.381453 28661 rpcz_store.cc:276] 0503 14:12:14.458950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.459169 (+ 219us) service_pool.cc:225] Handling call
0503 14:12:17.200786 (+2741617us) ts_tablet_manager.cc:1120] Deleting tablet 8c7d57fc4486457e9848daad79f04356
0503 14:12:17.200800 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.316715 (+115915us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.352348 (+ 35633us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.381243 (+ 28895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2736150}
I20240503 14:12:17.382012 28661 tablet_replica.cc:330] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.385214 28653 tablet_replica.cc:330] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:17.386301 28661 raft_consensus.cc:2229] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.386644 28661 raft_consensus.cc:2258] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.387077 28653 raft_consensus.cc:2229] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.389117 28653 raft_consensus.cc:2258] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.395237 29456 consensus_queue.cc:1035] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:17.404330 28653 ts_tablet_manager.cc:1875] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.409664 28661 ts_tablet_manager.cc:1875] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.410817 28600 tablet_bootstrap.cc:492] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.411098 28600 ts_tablet_manager.cc:1367] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.197s user 0.001s sys 0.006s
I20240503 14:12:17.411995 28600 raft_consensus.cc:348] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.412391 28600 raft_consensus.cc:374] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.412513 28600 raft_consensus.cc:729] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.412844 28600 consensus_queue.cc:260] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.417071 28600 ts_tablet_manager.cc:1398] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:17.418999 28264 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:17.427011 28600 tablet_bootstrap.cc:492] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.432967 28604 tablet_bootstrap.cc:492] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.433302 28604 ts_tablet_manager.cc:1367] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.069s user 0.016s sys 0.004s
I20240503 14:12:17.434432 28604 raft_consensus.cc:348] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.435117 28604 raft_consensus.cc:374] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.435240 28604 raft_consensus.cc:729] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.435564 28604 consensus_queue.cc:260] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.436419 28604 ts_tablet_manager.cc:1398] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:17.436795 28604 tablet_bootstrap.cc:492] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.438859 28604 tablet_bootstrap.cc:654] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.442348 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 16fed718124a4640967ab13a1ac2b379 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.443143 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d9f47543524fc5921c99aad869035f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.443755 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet e21a5dafa82f4660927d96c779a23478 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.444825 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 06fb79fcf30d485798c64c815e372dc1 (table default.Table_3668987e06c74fa6bc566d6e46f92879 [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
I20240503 14:12:17.445533 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 007c84fd9d9247728b0babd48689aba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4451a8c3ab1e4fccac7100ac74bb990f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.446691 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 3e81239aacd44a2aba8fa7b5d9d4d893 (DEFAULT_TABLE table=default.Table_df59445155274498a94afe77d8f56c14 [id=e332470c43ee41eab18e72561dd23723]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.447178 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e81239aacd44a2aba8fa7b5d9d4d893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.449517 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 843f0b87687340558d7c6acbbad2c4d7 (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.449873 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d9f47543524fc5921c99aad869035f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.449581 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet e21a5dafa82f4660927d96c779a23478 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.450418 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843f0b87687340558d7c6acbbad2c4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.455446 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 16fed718124a4640967ab13a1ac2b379 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.456179 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 3e81239aacd44a2aba8fa7b5d9d4d893 (DEFAULT_TABLE table=default.Table_df59445155274498a94afe77d8f56c14 [id=e332470c43ee41eab18e72561dd23723]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.460348 28653 ts_tablet_manager.cc:1888] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.460611 28653 log.cc:1196] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/daf5d153ab5142c0902cc6e96629f412
I20240503 14:12:17.461284 28653 ts_tablet_manager.cc:1909] T daf5d153ab5142c0902cc6e96629f412 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:17.462226 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 16fed718124a4640967ab13a1ac2b379 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.462464 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 007c84fd9d9247728b0babd48689aba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4451a8c3ab1e4fccac7100ac74bb990f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.462736 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d9f47543524fc5921c99aad869035f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.462846 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet e21a5dafa82f4660927d96c779a23478 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.462992 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 007c84fd9d9247728b0babd48689aba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4451a8c3ab1e4fccac7100ac74bb990f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.463490 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 8900cba02fdb46a9bb08943812d3709d (DEFAULT_TABLE table=default.Table_8fc00adb6ca146859c462157a163a3c9 [id=65ace760cd924ba08b7318175705182e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.463490 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 843f0b87687340558d7c6acbbad2c4d7 (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.463982 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8900cba02fdb46a9bb08943812d3709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.466857 29456 raft_consensus.cc:482] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.464319 28653 tablet_replica.cc:330] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:17.468705 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843f0b87687340558d7c6acbbad2c4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.468812 28653 raft_consensus.cc:2229] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.469478 28653 raft_consensus.cc:2258] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.471196 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 3e81239aacd44a2aba8fa7b5d9d4d893 (DEFAULT_TABLE table=default.Table_df59445155274498a94afe77d8f56c14 [id=e332470c43ee41eab18e72561dd23723]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.471760 28653 ts_tablet_manager.cc:1875] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.472010 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 843f0b87687340558d7c6acbbad2c4d7 (DEFAULT_TABLE table=default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.473501 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e81239aacd44a2aba8fa7b5d9d4d893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.444609 28661 ts_tablet_manager.cc:1888] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.474642 28661 log.cc:1196] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0ad1cd0e80cd4adbbbbcd60e335a5c49
I20240503 14:12:17.478754 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e0a149bc29c4f2fac86f41074f45afa" } new_table_name: "default.Table_facfcbde0b2740359615d4907998c7fa" modify_external_catalogs: true
I20240503 14:12:17.478668 28661 ts_tablet_manager.cc:1909] T 0ad1cd0e80cd4adbbbbcd60e335a5c49 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.477550 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_c2b8fa4eaa2946d8b7f1bd78f114e8fe [id=874fc82c70674371a8be82614bb6fcfb] alter complete (version 5)
I20240503 14:12:17.482048 28604 tablet_bootstrap.cc:492] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.482424 28604 ts_tablet_manager.cc:1367] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.003s
I20240503 14:12:17.483536 28604 raft_consensus.cc:348] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.483754 28604 raft_consensus.cc:374] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.483801 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843f0b87687340558d7c6acbbad2c4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.483812 28604 raft_consensus.cc:729] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.484167 28604 consensus_queue.cc:260] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.484784 28604 ts_tablet_manager.cc:1398] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:17.484942 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 566) took 3012 ms. Trace:
I20240503 14:12:17.485627 28604 tablet_bootstrap.cc:492] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.485360 28661 rpcz_store.cc:276] 0503 14:12:14.472559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.472831 (+ 272us) service_pool.cc:225] Handling call
0503 14:12:17.381953 (+2909122us) ts_tablet_manager.cc:1120] Deleting tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49
0503 14:12:17.381965 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.429669 (+ 47704us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.444478 (+ 14809us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.484759 (+ 40281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2907796}
I20240503 14:12:17.488466 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 567) took 3015 ms. Trace:
I20240503 14:12:17.488852 28604 tablet_bootstrap.cc:654] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.489073 28661 rpcz_store.cc:276] 0503 14:12:14.472732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.472969 (+ 237us) service_pool.cc:225] Handling call
0503 14:12:17.487246 (+3014277us) ts_tablet_manager.cc:1120] Deleting tablet 77f0081cc2cb48aaa368741109032fd1
0503 14:12:17.487318 (+ 72us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.488281 (+ 963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3002369}
I20240503 14:12:17.491544 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 60d4f5591b1f4826bc09e57094fb46bf (DEFAULT_TABLE table=default.Table_89b0f6a84c3140aca776c1da1137e09f [id=ddde4ae6b52e4419b8d2a742ce3bdfc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.492098 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d4f5591b1f4826bc09e57094fb46bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.491773 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 565) took 3019 ms. Trace:
I20240503 14:12:17.492213 28661 rpcz_store.cc:276] 0503 14:12:14.472407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.495409 (+ 23002us) service_pool.cc:225] Handling call
0503 14:12:17.490779 (+2995370us) ts_tablet_manager.cc:1120] Deleting tablet 920be6d87c4440d0bf7453c86cfe0b5c
0503 14:12:17.490790 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.491584 (+ 794us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2987315}
I20240503 14:12:17.495092 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 564) took 3023 ms. Trace:
I20240503 14:12:17.496296 28600 tablet_bootstrap.cc:654] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.495671 28661 rpcz_store.cc:276] 0503 14:12:14.471951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.499455 (+ 27504us) service_pool.cc:225] Handling call
0503 14:12:17.494126 (+2994671us) ts_tablet_manager.cc:1120] Deleting tablet c9c0540960ac45aabeaf13f888caf238
0503 14:12:17.494303 (+ 177us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.494912 (+ 609us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2985271}
I20240503 14:12:17.497565 28661 tablet_replica.cc:330] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.499002 28661 raft_consensus.cc:2229] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.499866 28661 raft_consensus.cc:2258] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.500314 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e81239aacd44a2aba8fa7b5d9d4d893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.503682 28661 ts_tablet_manager.cc:1875] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.504027 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 60d4f5591b1f4826bc09e57094fb46bf (DEFAULT_TABLE table=default.Table_89b0f6a84c3140aca776c1da1137e09f [id=ddde4ae6b52e4419b8d2a742ce3bdfc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.504858 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d4f5591b1f4826bc09e57094fb46bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.511118 28600 tablet_bootstrap.cc:492] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.511386 28600 ts_tablet_manager.cc:1367] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.002s
I20240503 14:12:17.512849 28653 ts_tablet_manager.cc:1888] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.513600 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba0bac965af4446b601ff70f474b3ab (DEFAULT_TABLE table=default.Table_84ebaac5471744d0ad8b983a03565cee [id=4bc361205572492d968f4ee9841ae645]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.514062 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba0bac965af4446b601ff70f474b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.514231 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba0bac965af4446b601ff70f474b3ab (DEFAULT_TABLE table=default.Table_84ebaac5471744d0ad8b983a03565cee [id=4bc361205572492d968f4ee9841ae645]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.514645 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba0bac965af4446b601ff70f474b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.515687 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 03700c1efc08471cb67dfe3a8016a740 (DEFAULT_TABLE table=default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.516103 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03700c1efc08471cb67dfe3a8016a740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.512379 28600 raft_consensus.cc:348] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:17.516932 28600 raft_consensus.cc:374] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.518319 28604 tablet_bootstrap.cc:492] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.518469 28604 ts_tablet_manager.cc:1367] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.009s
I20240503 14:12:17.518775 28600 raft_consensus.cc:729] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.519333 28604 raft_consensus.cc:348] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.519600 28604 raft_consensus.cc:374] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.519656 28604 raft_consensus.cc:729] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.519883 28604 consensus_queue.cc:260] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.520462 28604 ts_tablet_manager.cc:1398] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:17.520763 28604 tablet_bootstrap.cc:492] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.522639 28604 tablet_bootstrap.cc:654] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.522938 28600 consensus_queue.cc:260] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:17.524993 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 60d4f5591b1f4826bc09e57094fb46bf (DEFAULT_TABLE table=default.Table_89b0f6a84c3140aca776c1da1137e09f [id=ddde4ae6b52e4419b8d2a742ce3bdfc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.525533 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d4f5591b1f4826bc09e57094fb46bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.523571 28653 log.cc:1196] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2331aec871824a238f3f64de217d7092
I20240503 14:12:17.530862 28179 tablet_service.cc:1467] Processing CreateTablet for tablet e708be06de744743aac4e192bdd0f449 (DEFAULT_TABLE table=default.Table_a52e2109f2f847dc992dd85cf261b6e3 [id=d27fc3ac100145ad893262a35930c590]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.531800 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e708be06de744743aac4e192bdd0f449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.533253 28653 ts_tablet_manager.cc:1909] T 2331aec871824a238f3f64de217d7092 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:17.533885 28600 ts_tablet_manager.cc:1398] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 14:12:17.535892 28600 tablet_bootstrap.cc:492] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.467036 29456 raft_consensus.cc:504] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.539631 29598 raft_consensus.cc:482] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.539705 29456 leader_election.cc:288] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.539784 29598 raft_consensus.cc:504] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.542639 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65853212c2404597b015ddc8ea99ee47" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.542770 28336 tablet_service.cc:1467] Processing CreateTablet for tablet e708be06de744743aac4e192bdd0f449 (DEFAULT_TABLE table=default.Table_a52e2109f2f847dc992dd85cf261b6e3 [id=d27fc3ac100145ad893262a35930c590]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.543357 28324 tablet_service.cc:1467] Processing CreateTablet for tablet f3023e56a9c94e2792508d2e21fe353f (DEFAULT_TABLE table=default.Table_215b8963e0d64362a80720493fdbcdd0 [id=005840e9cc5f4245ba684696f2434604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.543836 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 03700c1efc08471cb67dfe3a8016a740 (DEFAULT_TABLE table=default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:17.543887 28290 leader_election.cc:341] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.544247 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65853212c2404597b015ddc8ea99ee47" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:17.544479 28214 raft_consensus.cc:2454] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:17.544736 29652 raft_consensus.cc:482] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.544868 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e708be06de744743aac4e192bdd0f449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.544875 29652 raft_consensus.cc:504] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.545490 29652 leader_election.cc:288] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.548651 28290 leader_election.cc:302] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.548995 29456 raft_consensus.cc:2781] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.549079 29456 raft_consensus.cc:482] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.549140 29456 raft_consensus.cc:3037] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.550415 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03700c1efc08471cb67dfe3a8016a740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.552448 28603 ts_tablet_manager.cc:1888] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:17.552573 28603 log.cc:1196] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/06fb79fcf30d485798c64c815e372dc1
I20240503 14:12:17.552930 28603 ts_tablet_manager.cc:1909] T 06fb79fcf30d485798c64c815e372dc1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:17.552934 28661 ts_tablet_manager.cc:1888] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.553040 28661 log.cc:1196] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0bd5234791524eddb7edd493e39fb993
I20240503 14:12:17.553319 28661 ts_tablet_manager.cc:1909] T 0bd5234791524eddb7edd493e39fb993 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.554118 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 568) took 2945 ms. Trace:
I20240503 14:12:17.554217 28661 rpcz_store.cc:276] 0503 14:12:14.616444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.617254 (+ 810us) service_pool.cc:225] Handling call
0503 14:12:17.497336 (+2880082us) ts_tablet_manager.cc:1120] Deleting tablet 0bd5234791524eddb7edd493e39fb993
0503 14:12:17.497348 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.541076 (+ 43728us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.552830 (+ 11754us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.554090 (+ 1260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2870572}
I20240503 14:12:17.554682 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 577) took 2888 ms. Trace:
I20240503 14:12:17.554761 28661 rpcz_store.cc:276] 0503 14:12:14.665890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.666043 (+ 153us) service_pool.cc:225] Handling call
0503 14:12:17.554517 (+2888474us) ts_tablet_manager.cc:1120] Deleting tablet 2221218847554cbd9d3242a9a6431e46
0503 14:12:17.554527 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.554664 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2885614}
I20240503 14:12:17.555056 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 578) took 2770 ms. Trace:
I20240503 14:12:17.555124 28661 rpcz_store.cc:276] 0503 14:12:14.784239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.785846 (+ 1607us) service_pool.cc:225] Handling call
0503 14:12:17.554961 (+2769115us) ts_tablet_manager.cc:1120] Deleting tablet ea261d8d5d174bee804aa2e171f35008
0503 14:12:17.554969 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.555043 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2755571}
I20240503 14:12:17.555395 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 586) took 2713 ms. Trace:
I20240503 14:12:17.555467 28661 rpcz_store.cc:276] 0503 14:12:14.842085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.877160 (+ 35075us) service_pool.cc:225] Handling call
0503 14:12:17.555310 (+2678150us) ts_tablet_manager.cc:1120] Deleting tablet 3cbcae22ea124edaa14c9409f3052714
0503 14:12:17.555317 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.555383 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672917}
I20240503 14:12:17.555747 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 587) took 2708 ms. Trace:
I20240503 14:12:17.555817 28661 rpcz_store.cc:276] 0503 14:12:14.847440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:14.882436 (+ 34996us) service_pool.cc:225] Handling call
0503 14:12:17.555658 (+2673222us) ts_tablet_manager.cc:1120] Deleting tablet 95c97a43bd714da68c1a77408f3d1cd1
0503 14:12:17.555666 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.555735 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672886}
I20240503 14:12:17.559715 28600 tablet_bootstrap.cc:654] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.561443 29456 raft_consensus.cc:504] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.562198 29456 leader_election.cc:288] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.563239 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65853212c2404597b015ddc8ea99ee47" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:17.563329 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65853212c2404597b015ddc8ea99ee47" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:17.563503 28214 raft_consensus.cc:3037] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:17.563943 28290 leader_election.cc:341] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.564415 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3023e56a9c94e2792508d2e21fe353f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.553278 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 657) took 1057 ms. Trace:
I20240503 14:12:17.566047 28603 rpcz_store.cc:276] 0503 14:12:16.495794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.495875 (+ 81us) service_pool.cc:225] Handling call
0503 14:12:17.300980 (+805105us) ts_tablet_manager.cc:1120] Deleting tablet 06fb79fcf30d485798c64c815e372dc1
0503 14:12:17.300992 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.452329 (+151337us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.552342 (+100013us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.553253 (+ 911us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":804958}
I20240503 14:12:17.566533 28603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 658) took 1017 ms. Trace:
I20240503 14:12:17.566592 28214 raft_consensus.cc:2454] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:17.566614 28603 rpcz_store.cc:276] 0503 14:12:16.549239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.549302 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:17.566396 (+1017094us) ts_tablet_manager.cc:1120] Deleting tablet f080c95ff5ca41e0b3d27382040107e9
0503 14:12:17.566406 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.566511 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":953554}
I20240503 14:12:17.567600 28290 leader_election.cc:302] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.568198 29456 raft_consensus.cc:2781] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.568324 29456 raft_consensus.cc:686] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:17.568571 29456 consensus_queue.cc:237] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.571070 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 588) took 2525 ms. Trace:
I20240503 14:12:17.571189 28661 rpcz_store.cc:276] 0503 14:12:15.045968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.047318 (+ 1350us) service_pool.cc:225] Handling call
0503 14:12:17.555992 (+2508674us) ts_tablet_manager.cc:1120] Deleting tablet ff1958dc488e4854a8c8e74da9fab243
0503 14:12:17.556000 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.571042 (+ 15042us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2508416}
I20240503 14:12:17.571610 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 589) took 2525 ms. Trace:
I20240503 14:12:17.571687 28661 rpcz_store.cc:276] 0503 14:12:15.046119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.055438 (+ 9319us) service_pool.cc:225] Handling call
0503 14:12:17.571489 (+2516051us) ts_tablet_manager.cc:1120] Deleting tablet b59945d011b64cba87c854e2ffde8226
0503 14:12:17.571498 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.571594 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2505859}
I20240503 14:12:17.571964 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 590) took 2525 ms. Trace:
I20240503 14:12:17.572002 28604 tablet_bootstrap.cc:492] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.572306 28604 ts_tablet_manager.cc:1367] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.006s
I20240503 14:12:17.573361 28604 raft_consensus.cc:348] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.579169 28603 tablet_replica.cc:330] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:17.579468 28603 raft_consensus.cc:2229] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:17.579999 28603 raft_consensus.cc:2258] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.580951 28603 ts_tablet_manager.cc:1875] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.584256 28604 raft_consensus.cc:374] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.584363 28604 raft_consensus.cc:729] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.572031 28661 rpcz_store.cc:276] 0503 14:12:15.046255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.057663 (+ 11408us) service_pool.cc:225] Handling call
0503 14:12:17.571875 (+2514212us) ts_tablet_manager.cc:1120] Deleting tablet f0287bc315644f0aaa976a56174576d9
0503 14:12:17.571883 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.571952 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506119}
I20240503 14:12:17.585289 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 598) took 2362 ms. Trace:
I20240503 14:12:17.585558 28661 rpcz_store.cc:276] 0503 14:12:15.222219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.222289 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:17.584856 (+2362567us) ts_tablet_manager.cc:1120] Deleting tablet 2a90167ae51647688dfc502b00c5c9c6
0503 14:12:17.584871 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.584988 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352470}
I20240503 14:12:17.586791 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 599) took 2361 ms. Trace:
I20240503 14:12:17.587308 28661 rpcz_store.cc:276] 0503 14:12:15.225145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.225209 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:17.586598 (+2361389us) ts_tablet_manager.cc:1120] Deleting tablet 2cad43f3c6094253ba87d0ed3fc29fd9
0503 14:12:17.586608 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.586771 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2353249}
I20240503 14:12:17.587942 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 601) took 2352 ms. Trace:
I20240503 14:12:17.588366 28661 rpcz_store.cc:276] 0503 14:12:15.235405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.235869 (+ 464us) service_pool.cc:225] Handling call
0503 14:12:17.587825 (+2351956us) ts_tablet_manager.cc:1120] Deleting tablet 9e9d878dfb144e5db914f8619c13ce8c
0503 14:12:17.587835 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.587924 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2335620}
I20240503 14:12:17.588935 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 600) took 2353 ms. Trace:
I20240503 14:12:17.589345 28661 rpcz_store.cc:276] 0503 14:12:15.235261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.235911 (+ 650us) service_pool.cc:225] Handling call
0503 14:12:17.588829 (+2352918us) ts_tablet_manager.cc:1120] Deleting tablet 5f6c2307b8e64c2481d5bd47e9189fea
0503 14:12:17.588838 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.588920 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2336431}
I20240503 14:12:17.589905 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 603) took 2354 ms. Trace:
I20240503 14:12:17.590313 28661 rpcz_store.cc:276] 0503 14:12:15.235610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.235833 (+ 223us) service_pool.cc:225] Handling call
0503 14:12:17.589799 (+2353966us) ts_tablet_manager.cc:1120] Deleting tablet 77f6fb009bdd4f70ba90e7b5f972cea9
0503 14:12:17.589811 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.589889 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2329333}
I20240503 14:12:17.590883 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 602) took 2355 ms. Trace:
I20240503 14:12:17.586207 28604 consensus_queue.cc:260] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.591287 28661 rpcz_store.cc:276] 0503 14:12:15.235515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.235798 (+ 283us) service_pool.cc:225] Handling call
0503 14:12:17.590779 (+2354981us) ts_tablet_manager.cc:1120] Deleting tablet d3943c8358a3484d943bad0c906a9425
0503 14:12:17.590788 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.590865 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2330271}
I20240503 14:12:17.592675 28661 tablet_replica.cc:330] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.593132 28661 raft_consensus.cc:2229] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.593703 28661 raft_consensus.cc:2258] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.594691 28661 ts_tablet_manager.cc:1875] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.592013 28604 ts_tablet_manager.cc:1398] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240503 14:12:17.596405 28604 tablet_bootstrap.cc:492] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.600157 28604 tablet_bootstrap.cc:654] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.607724 28604 tablet_bootstrap.cc:492] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.608443 28604 ts_tablet_manager.cc:1367] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20240503 14:12:17.609520 28604 raft_consensus.cc:348] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.612323 28600 tablet_bootstrap.cc:492] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.613389 28604 raft_consensus.cc:374] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.613719 28604 raft_consensus.cc:729] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.613652 28600 ts_tablet_manager.cc:1367] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.002s
I20240503 14:12:17.614178 28604 consensus_queue.cc:260] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.615345 28604 ts_tablet_manager.cc:1398] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:17.615568 29652 raft_consensus.cc:482] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.615707 29652 raft_consensus.cc:504] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.616348 29652 leader_election.cc:288] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.617241 28604 tablet_bootstrap.cc:492] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.619194 28604 tablet_bootstrap.cc:654] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.620915 28614 tablet_bootstrap.cc:492] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:17.621155 28614 ts_tablet_manager.cc:1367] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.336s user 0.014s sys 0.018s
I20240503 14:12:17.621902 28614 raft_consensus.cc:348] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.622174 28614 raft_consensus.cc:374] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.621886 28661 ts_tablet_manager.cc:1888] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:12:17.622290 28661 log.cc:1196] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/cc230b489ab8455c8cb2eba31084fddb
I20240503 14:12:17.622303 28614 raft_consensus.cc:729] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.622547 28614 consensus_queue.cc:260] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.620962 28600 raft_consensus.cc:348] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:17.623008 28661 ts_tablet_manager.cc:1909] T cc230b489ab8455c8cb2eba31084fddb P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.623142 28614 ts_tablet_manager.cc:1398] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:17.622763 28600 raft_consensus.cc:374] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.623445 28614 tablet_bootstrap.cc:492] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:17.623616 28600 raft_consensus.cc:729] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.624773 28614 tablet_bootstrap.cc:654] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.625311 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 604) took 2344 ms. Trace:
I20240503 14:12:17.625458 28661 rpcz_store.cc:276] 0503 14:12:15.280914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.280992 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:17.592626 (+2311634us) ts_tablet_manager.cc:1120] Deleting tablet cc230b489ab8455c8cb2eba31084fddb
0503 14:12:17.592635 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.610436 (+ 17801us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.621767 (+ 11331us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.625284 (+ 3517us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2311375}
I20240503 14:12:17.632244 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 612) took 2090 ms. Trace:
I20240503 14:12:17.632476 28661 rpcz_store.cc:276] 0503 14:12:15.541806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.544133 (+ 2327us) service_pool.cc:225] Handling call
0503 14:12:17.632063 (+2087930us) ts_tablet_manager.cc:1120] Deleting tablet 79d09bee02c14822aa08050eac5be181
0503 14:12:17.632077 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.632219 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2087629}
I20240503 14:12:17.632871 28661 tablet_replica.cc:330] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.633158 28661 raft_consensus.cc:2229] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:17.633486 28661 pending_rounds.cc:62] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:17.633836 28661 pending_rounds.cc:68] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597713957515264 op_type: NO_OP noop_request { }
I20240503 14:12:17.634020 28661 raft_consensus.cc:2866] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:17.634174 28661 raft_consensus.cc:2258] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.635043 28661 ts_tablet_manager.cc:1875] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.625977 28600 consensus_queue.cc:260] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:17.639218 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba0bac965af4446b601ff70f474b3ab (DEFAULT_TABLE table=default.Table_84ebaac5471744d0ad8b983a03565cee [id=4bc361205572492d968f4ee9841ae645]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.639504 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e680f189add4b5c96de4046321aad2d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.639665 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba0bac965af4446b601ff70f474b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.639727 28352 raft_consensus.cc:2454] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:17.642282 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d9985eecda4f3194c3251216aa58ee" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.642972 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcfe8fff82a4fdd8c01a24a207a27ab (DEFAULT_TABLE table=default.Table_07fdeadf61a045d8aeb377c2a3c4c1af [id=be2dbf4734c74e93ad430d203979e633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.643352 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcfe8fff82a4fdd8c01a24a207a27ab (DEFAULT_TABLE table=default.Table_07fdeadf61a045d8aeb377c2a3c4c1af [id=be2dbf4734c74e93ad430d203979e633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.643378 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e680f189add4b5c96de4046321aad2d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:17.643585 28214 raft_consensus.cc:2454] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:17.644043 28184 tablet_service.cc:1467] Processing CreateTablet for tablet bf8d2474104a41a989cd7bd4480367f5 (DEFAULT_TABLE table=default.Table_2c7ddd58b18f4c80adef853298b2019e [id=7c4dfb9fe5944829b1b46908330bf37c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.644281 28600 ts_tablet_manager.cc:1398] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240503 14:12:17.648736 28604 tablet_bootstrap.cc:492] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.648885 28604 ts_tablet_manager.cc:1367] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20240503 14:12:17.650959 28328 tablet_service.cc:1467] Processing CreateTablet for tablet bf8d2474104a41a989cd7bd4480367f5 (DEFAULT_TABLE table=default.Table_2c7ddd58b18f4c80adef853298b2019e [id=7c4dfb9fe5944829b1b46908330bf37c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.654592 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 06fb79fcf30d485798c64c815e372dc1 (table default.Table_3668987e06c74fa6bc566d6e46f92879 [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
W20240503 14:12:17.655167 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f080c95ff5ca41e0b3d27382040107e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f080c95ff5ca41e0b3d27382040107e9 already in progress: creating tablet
W20240503 14:12:17.655855 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 07354861ee6142d6b1fbe573397d3160 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07354861ee6142d6b1fbe573397d3160 already in progress: creating tablet
W20240503 14:12:17.656605 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0f75e102dd4249c3800e1fcb4339acf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f75e102dd4249c3800e1fcb4339acf3 already in progress: creating tablet
W20240503 14:12:17.657148 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 62e4f6e3908e420e9fa156ee24b437d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e4f6e3908e420e9fa156ee24b437d6 already in progress: creating tablet
W20240503 14:12:17.658264 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1292a339f00247369ebeaa35752d51f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1292a339f00247369ebeaa35752d51f8 already in progress: creating tablet
I20240503 14:12:17.664609 28661 ts_tablet_manager.cc:1888] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:17.664842 28661 log.cc:1196] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a6dbb59d98834ab6bbac4abfa92681e3
I20240503 14:12:17.665331 28661 ts_tablet_manager.cc:1909] T a6dbb59d98834ab6bbac4abfa92681e3 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.669396 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 614) took 2127 ms. Trace:
I20240503 14:12:17.675269 28603 ts_tablet_manager.cc:1888] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:17.675582 28603 log.cc:1196] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0926e523ef454a2089f4d40a328c1c6d
I20240503 14:12:17.675911 28661 rpcz_store.cc:276] 0503 14:12:15.542066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.549496 (+ 7430us) service_pool.cc:225] Handling call
0503 14:12:17.632825 (+2083329us) ts_tablet_manager.cc:1120] Deleting tablet a6dbb59d98834ab6bbac4abfa92681e3
0503 14:12:17.632834 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.645073 (+ 12239us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.664472 (+ 19399us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.669284 (+ 4812us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2083062}
I20240503 14:12:17.676272 28603 ts_tablet_manager.cc:1909] T 0926e523ef454a2089f4d40a328c1c6d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:17.676460 28661 tablet_replica.cc:330] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.676929 28187 tablet_service.cc:1467] Processing CreateTablet for tablet f3023e56a9c94e2792508d2e21fe353f (DEFAULT_TABLE table=default.Table_215b8963e0d64362a80720493fdbcdd0 [id=005840e9cc5f4245ba684696f2434604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.677189 28661 raft_consensus.cc:2229] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:17.678665 28661 pending_rounds.cc:62] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:17.679333 28661 pending_rounds.cc:68] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597713960685568 op_type: NO_OP noop_request { }
I20240503 14:12:17.679539 28661 raft_consensus.cc:2866] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:17.679688 28661 raft_consensus.cc:2258] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.680661 28661 ts_tablet_manager.cc:1875] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.681762 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 648) took 1022 ms. Trace:
I20240503 14:12:17.681885 28653 rpcz_store.cc:276] 0503 14:12:16.658885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.659086 (+ 201us) service_pool.cc:225] Handling call
0503 14:12:17.539995 (+880909us) ts_tablet_manager.cc:1120] Deleting tablet 0f75e102dd4249c3800e1fcb4339acf3
0503 14:12:17.681547 (+141552us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.681736 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":880267}
I20240503 14:12:17.683867 29602 raft_consensus.cc:482] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.684032 29602 raft_consensus.cc:504] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.684804 29602 leader_election.cc:288] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.685760 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.685758 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:17.686339 28150 leader_election.cc:341] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.686559 28150 leader_election.cc:341] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.686630 28150 leader_election.cc:302] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.687196 29602 raft_consensus.cc:2726] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.688373 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcfe8fff82a4fdd8c01a24a207a27ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.688555 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8d2474104a41a989cd7bd4480367f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.690624 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 650) took 1030 ms. Trace:
I20240503 14:12:17.690716 28653 rpcz_store.cc:276] 0503 14:12:16.659985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.660032 (+ 47us) service_pool.cc:225] Handling call
0503 14:12:17.682217 (+1022185us) ts_tablet_manager.cc:1120] Deleting tablet 1292a339f00247369ebeaa35752d51f8
0503 14:12:17.690440 (+ 8223us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.690600 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022001}
I20240503 14:12:17.691049 28653 tablet_replica.cc:330] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:17.691223 28653 raft_consensus.cc:2229] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.691370 28653 raft_consensus.cc:2258] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.691918 28653 ts_tablet_manager.cc:1875] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.692154 28600 tablet_bootstrap.cc:492] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.696684 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3023e56a9c94e2792508d2e21fe353f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.700356 28600 tablet_bootstrap.cc:654] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.700003 28604 raft_consensus.cc:348] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.701282 28604 raft_consensus.cc:374] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.701884 29456 raft_consensus.cc:482] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.702044 29456 raft_consensus.cc:504] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.701694 28604 raft_consensus.cc:729] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.702193 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcfe8fff82a4fdd8c01a24a207a27ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.702772 29456 leader_election.cc:288] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.703106 28604 consensus_queue.cc:260] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:17.704195 28290 leader_election.cc:341] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 8b1728b6f0e64661aaa8184ce51366d2
I20240503 14:12:17.704488 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8d2474104a41a989cd7bd4480367f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.706427 28604 ts_tablet_manager.cc:1398] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:12:17.707515 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1728b6f0e64661aaa8184ce51366d2" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.707736 28493 raft_consensus.cc:2454] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:17.708015 28604 tablet_bootstrap.cc:492] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.708263 28290 leader_election.cc:302] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:17.708750 29456 raft_consensus.cc:2781] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.672585 28427 leader_election.cc:302] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:17.708834 29456 raft_consensus.cc:482] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.708894 29456 raft_consensus.cc:3037] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.709110 29652 raft_consensus.cc:2781] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.709210 29652 raft_consensus.cc:482] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.709293 29652 raft_consensus.cc:3037] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:17.709798 28427 leader_election.cc:341] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.711696 29652 raft_consensus.cc:504] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.712460 28472 tablet_service.cc:1467] Processing CreateTablet for tablet f3023e56a9c94e2792508d2e21fe353f (DEFAULT_TABLE table=default.Table_215b8963e0d64362a80720493fdbcdd0 [id=005840e9cc5f4245ba684696f2434604]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:17.712635 28427 leader_election.cc:341] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 95d9985eecda4f3194c3251216aa58ee
I20240503 14:12:17.712862 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3023e56a9c94e2792508d2e21fe353f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.712833 28427 leader_election.cc:302] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.713171 29516 raft_consensus.cc:2726] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.716388 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 03700c1efc08471cb67dfe3a8016a740 (DEFAULT_TABLE table=default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.716807 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03700c1efc08471cb67dfe3a8016a740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.717427 29652 leader_election.cc:288] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.720453 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcfe8fff82a4fdd8c01a24a207a27ab (DEFAULT_TABLE table=default.Table_07fdeadf61a045d8aeb377c2a3c4c1af [id=be2dbf4734c74e93ad430d203979e633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.720850 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcfe8fff82a4fdd8c01a24a207a27ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.721335 29602 raft_consensus.cc:482] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.721486 29602 raft_consensus.cc:504] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.722162 29602 leader_election.cc:288] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.723271 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cdd090bf0f493f8b92341369378532" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.723481 28493 raft_consensus.cc:2454] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:17.724042 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cdd090bf0f493f8b92341369378532" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:17.724792 28150 leader_election.cc:341] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.725009 28150 leader_election.cc:302] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.725062 28462 tablet_service.cc:1467] Processing CreateTablet for tablet e708be06de744743aac4e192bdd0f449 (DEFAULT_TABLE table=default.Table_a52e2109f2f847dc992dd85cf261b6e3 [id=d27fc3ac100145ad893262a35930c590]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.725456 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e708be06de744743aac4e192bdd0f449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.725488 29602 raft_consensus.cc:2781] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.725569 29602 raft_consensus.cc:482] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.725646 29602 raft_consensus.cc:3037] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.726846 29456 raft_consensus.cc:504] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:17.727793 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f6061918660e4070a0c8339cb5009ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6061918660e4070a0c8339cb5009ff8 already in progress: creating tablet
I20240503 14:12:17.716388 28469 tablet_service.cc:1467] Processing CreateTablet for tablet bf8d2474104a41a989cd7bd4480367f5 (DEFAULT_TABLE table=default.Table_2c7ddd58b18f4c80adef853298b2019e [id=7c4dfb9fe5944829b1b46908330bf37c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.728796 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 8c7d57fc4486457e9848daad79f04356 (table default.Table_ea4ed2f332dc46aaa1f6e7025a1a1c3c [id=5cf73f88a73a4cc5966467ecfff2140f]) successfully deleted
I20240503 14:12:17.728643 29602 raft_consensus.cc:504] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.729252 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8d2474104a41a989cd7bd4480367f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.729547 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0ad1cd0e80cd4adbbbbcd60e335a5c49 (table default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]) successfully deleted
W20240503 14:12:17.729986 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 77f0081cc2cb48aaa368741109032fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f0081cc2cb48aaa368741109032fd1 already in progress: creating tablet
I20240503 14:12:17.730607 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cdd090bf0f493f8b92341369378532" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:17.732172 29595 raft_consensus.cc:482] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.732321 29595 raft_consensus.cc:504] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:17.732506 28290 leader_election.cc:341] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 8b1728b6f0e64661aaa8184ce51366d2
I20240503 14:12:17.732934 29595 leader_election.cc:288] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.733264 29652 raft_consensus.cc:482] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.733448 29652 raft_consensus.cc:504] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.734038 29652 leader_election.cc:288] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.734252 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.734488 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:17.734676 28214 raft_consensus.cc:2454] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:17.735143 28150 leader_election.cc:341] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.735189 28290 leader_election.cc:341] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.735569 28290 leader_election.cc:302] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.736330 28653 ts_tablet_manager.cc:1888] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.736512 28653 log.cc:1196] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2404807f8500437ababca53f6388b636
I20240503 14:12:17.736579 29595 raft_consensus.cc:2781] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.736665 29595 raft_consensus.cc:482] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.736725 29595 raft_consensus.cc:3037] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.737232 28653 ts_tablet_manager.cc:1909] T 2404807f8500437ababca53f6388b636 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:17.737341 28603 tablet_replica.cc:330] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:17.737527 28603 raft_consensus.cc:2229] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.737713 28603 raft_consensus.cc:2258] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.737978 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cdd090bf0f493f8b92341369378532" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:17.738175 28493 raft_consensus.cc:3037] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.738276 28603 ts_tablet_manager.cc:1875] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.739581 29595 raft_consensus.cc:504] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.740739 28493 raft_consensus.cc:2454] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:17.741443 28150 leader_election.cc:302] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.742040 29595 leader_election.cc:288] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.741914 29602 leader_election.cc:288] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.742451 29602 raft_consensus.cc:2781] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.742426 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:17.742640 28214 raft_consensus.cc:3037] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.742661 29602 raft_consensus.cc:686] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:17.742954 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:17.743628 28290 leader_election.cc:341] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.744663 28661 ts_tablet_manager.cc:1888] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:17.744777 28661 log.cc:1196] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ed9262a7e828489ab51a745068c737df
I20240503 14:12:17.745128 28661 ts_tablet_manager.cc:1909] T ed9262a7e828489ab51a745068c737df P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.745124 28214 raft_consensus.cc:2454] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:17.745735 28290 leader_election.cc:302] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.746022 29595 raft_consensus.cc:2781] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.745298 29602 consensus_queue.cc:237] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.746567 28653 tablet_replica.cc:330] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:17.747732 28653 raft_consensus.cc:2229] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.748940 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e680f189add4b5c96de4046321aad2d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:17.749131 28352 raft_consensus.cc:3037] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.749101 29456 leader_election.cc:288] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.752274 28352 raft_consensus.cc:2454] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:17.752832 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1728b6f0e64661aaa8184ce51366d2" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:17.753170 28493 raft_consensus.cc:3037] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.748680 28653 raft_consensus.cc:2258] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.753615 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d4e4a1f0b14406be957426a45bf45d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.749806 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e680f189add4b5c96de4046321aad2d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:17.754191 28214 raft_consensus.cc:3037] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.755750 28600 tablet_bootstrap.cc:492] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.755970 28600 ts_tablet_manager.cc:1367] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.003s
I20240503 14:12:17.756237 28493 raft_consensus.cc:2454] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:17.756685 28653 ts_tablet_manager.cc:1875] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.756783 28290 leader_election.cc:302] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:17.757016 28214 raft_consensus.cc:2454] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:17.752811 28604 tablet_bootstrap.cc:654] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:17.749142 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 920be6d87c4440d0bf7453c86cfe0b5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920be6d87c4440d0bf7453c86cfe0b5c already in progress: creating tablet
I20240503 14:12:17.757285 29456 raft_consensus.cc:2781] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.757392 29456 raft_consensus.cc:686] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
W20240503 14:12:17.757555 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c9c0540960ac45aabeaf13f888caf238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c0540960ac45aabeaf13f888caf238 already in progress: creating tablet
I20240503 14:12:17.757633 29456 consensus_queue.cc:237] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.758213 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0bd5234791524eddb7edd493e39fb993 (table default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]) successfully deleted
W20240503 14:12:17.758597 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2221218847554cbd9d3242a9a6431e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2221218847554cbd9d3242a9a6431e46 already in progress: creating tablet
I20240503 14:12:17.758829 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 615) took 2216 ms. Trace:
W20240503 14:12:17.758885 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ea261d8d5d174bee804aa2e171f35008 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea261d8d5d174bee804aa2e171f35008 already in progress: creating tablet
I20240503 14:12:17.758922 28661 rpcz_store.cc:276] 0503 14:12:15.542216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.549799 (+ 7583us) service_pool.cc:225] Handling call
0503 14:12:17.676401 (+2126602us) ts_tablet_manager.cc:1120] Deleting tablet ed9262a7e828489ab51a745068c737df
0503 14:12:17.676413 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.697074 (+ 20661us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.744533 (+ 47459us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.758802 (+ 14269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2126409}
W20240503 14:12:17.759208 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 3cbcae22ea124edaa14c9409f3052714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cbcae22ea124edaa14c9409f3052714 already in progress: creating tablet
W20240503 14:12:17.759516 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 95c97a43bd714da68c1a77408f3d1cd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c97a43bd714da68c1a77408f3d1cd1 already in progress: creating tablet
W20240503 14:12:17.759974 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ff1958dc488e4854a8c8e74da9fab243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1958dc488e4854a8c8e74da9fab243 already in progress: creating tablet
I20240503 14:12:17.760330 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 613) took 2218 ms. Trace:
W20240503 14:12:17.760409 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b59945d011b64cba87c854e2ffde8226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b59945d011b64cba87c854e2ffde8226 already in progress: creating tablet
W20240503 14:12:17.760790 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f0287bc315644f0aaa976a56174576d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0287bc315644f0aaa976a56174576d9 already in progress: creating tablet
I20240503 14:12:17.761323 28661 rpcz_store.cc:276] 0503 14:12:15.541925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.552319 (+ 10394us) service_pool.cc:225] Handling call
0503 14:12:17.760171 (+2207852us) ts_tablet_manager.cc:1120] Deleting tablet d43823da09bb49be8f5d1a1149a4b11d
0503 14:12:17.760186 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.760307 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2207544}
I20240503 14:12:17.761617 28600 raft_consensus.cc:348] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:17.762245 28661 tablet_replica.cc:330] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.762766 28600 raft_consensus.cc:374] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.762995 28600 raft_consensus.cc:729] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.763017 28661 raft_consensus.cc:2229] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.763360 28600 consensus_queue.cc:260] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:17.763793 28661 raft_consensus.cc:2258] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.764480 28661 ts_tablet_manager.cc:1875] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.764957 28600 ts_tablet_manager.cc:1398] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:12:17.766008 28600 tablet_bootstrap.cc:492] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.766582 29595 raft_consensus.cc:686] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:17.767123 29595 consensus_queue.cc:237] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.770939 28600 tablet_bootstrap.cc:654] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.764897 28427 leader_election.cc:302] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:17.769057 29652 raft_consensus.cc:482] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.772337 29652 raft_consensus.cc:504] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.773095 29652 leader_election.cc:288] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.773516 29652 raft_consensus.cc:2781] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.773653 29652 raft_consensus.cc:686] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:17.773911 29652 consensus_queue.cc:237] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.775400 28614 tablet_bootstrap.cc:492] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
W20240503 14:12:17.777343 28427 leader_election.cc:341] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.778865 28427 leader_election.cc:341] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b7d4e4a1f0b14406be957426a45bf45d
W20240503 14:12:17.779012 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2a90167ae51647688dfc502b00c5c9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a90167ae51647688dfc502b00c5c9c6 already in progress: creating tablet
I20240503 14:12:17.779597 28427 leader_election.cc:302] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.779989 29652 raft_consensus.cc:2726] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:17.780673 28427 leader_election.cc:341] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2404807f8500437ababca53f6388b636
W20240503 14:12:17.780896 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9e9d878dfb144e5db914f8619c13ce8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9d878dfb144e5db914f8619c13ce8c already in progress: creating tablet
I20240503 14:12:17.781652 29602 raft_consensus.cc:482] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.783093 29652 consensus_queue.cc:1035] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:17.788509 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5f6c2307b8e64c2481d5bd47e9189fea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f6c2307b8e64c2481d5bd47e9189fea already in progress: creating tablet
I20240503 14:12:17.781929 28600 tablet_bootstrap.cc:492] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.789208 28600 ts_tablet_manager.cc:1367] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240503 14:12:17.778532 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2404807f8500437ababca53f6388b636" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.782011 29602 raft_consensus.cc:504] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:17.790518 28614 ts_tablet_manager.cc:1367] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.167s user 0.006s sys 0.026s
I20240503 14:12:17.790620 28661 ts_tablet_manager.cc:1888] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.791538 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:17.791826 28217 raft_consensus.cc:2454] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:17.792399 28290 leader_election.cc:302] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:17.792786 29456 raft_consensus.cc:2781] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.792917 29456 raft_consensus.cc:482] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.793077 29456 raft_consensus.cc:3037] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.793438 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5297664a4d943c28adfb6ac4f061680" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:17.794649 28150 leader_election.cc:341] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.795019 29602 leader_election.cc:288] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:17.795496 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 77f6fb009bdd4f70ba90e7b5f972cea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f6fb009bdd4f70ba90e7b5f972cea9 already in progress: creating tablet
I20240503 14:12:17.793550 29598 leader_election.cc:288] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.791988 28614 raft_consensus.cc:348] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.795986 28614 raft_consensus.cc:374] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:12:17.796142 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d3943c8358a3484d943bad0c906a9425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3943c8358a3484d943bad0c906a9425 already in progress: creating tablet
I20240503 14:12:17.796197 28614 raft_consensus.cc:729] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.796489 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet cc230b489ab8455c8cb2eba31084fddb (table default.Table_19f17d20edd24196b9efade6dca84369 [id=084b186eba154ce1939f530379f4ce76]) successfully deleted
I20240503 14:12:17.796489 28614 consensus_queue.cc:260] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.790243 28600 raft_consensus.cc:348] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.796959 28600 raft_consensus.cc:374] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.796890 29456 raft_consensus.cc:504] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.797099 28614 ts_tablet_manager.cc:1398] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:17.797425 28614 tablet_bootstrap.cc:492] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:17.798120 29456 leader_election.cc:288] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.798043 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:17.801587 28217 raft_consensus.cc:3037] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.804863 28217 raft_consensus.cc:2454] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:17.805430 28600 raft_consensus.cc:729] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.805433 28290 leader_election.cc:302] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:17.805730 29598 raft_consensus.cc:2781] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.805871 29598 raft_consensus.cc:686] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
W20240503 14:12:17.797461 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 79d09bee02c14822aa08050eac5be181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79d09bee02c14822aa08050eac5be181 already in progress: creating tablet
I20240503 14:12:17.806133 29598 consensus_queue.cc:237] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.806488 28600 consensus_queue.cc:260] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.806969 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a6dbb59d98834ab6bbac4abfa92681e3 (table default.Table_33f62889c4784d6cb42ac76faa20113c [id=c62152292eb14f4aaae391155d12da6f]) successfully deleted
I20240503 14:12:17.807720 28600 ts_tablet_manager.cc:1398] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 14:12:17.808900 28600 tablet_bootstrap.cc:492] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.809201 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ed9262a7e828489ab51a745068c737df (table default.Table_dbb71b03f52c48d29206e945b956f702 [id=fd3e3a26ac344865b4c622ebe130ebff]) successfully deleted
I20240503 14:12:17.810075 28614 tablet_bootstrap.cc:654] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:17.810489 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d43823da09bb49be8f5d1a1149a4b11d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43823da09bb49be8f5d1a1149a4b11d already in progress: creating tablet
I20240503 14:12:17.809731 28603 ts_tablet_manager.cc:1888] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:17.811267 28603 log.cc:1196] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a32b6995818041778617a0d6bf936c8c
I20240503 14:12:17.811666 28603 ts_tablet_manager.cc:1909] T a32b6995818041778617a0d6bf936c8c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:17.812165 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet daf5d153ab5142c0902cc6e96629f412 (table default.Table_6673b8d0a18d42c3a18f6e611c596580 [id=3c43a0fcf51143179a1ed0e1b8e333c1]) successfully deleted
I20240503 14:12:17.813499 28600 tablet_bootstrap.cc:654] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.792907 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5297664a4d943c28adfb6ac4f061680" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.794124 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.798636 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:17.814071 28661 log.cc:1196] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/962df44408484e3f8d7b7138f27c1a66
W20240503 14:12:17.814805 28290 leader_election.cc:341] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.815186 28290 leader_election.cc:341] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.821444 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 07354861ee6142d6b1fbe573397d3160 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07354861ee6142d6b1fbe573397d3160 already in progress: creating tablet
I20240503 14:12:17.824282 28661 ts_tablet_manager.cc:1909] T 962df44408484e3f8d7b7138f27c1a66 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.825428 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 616) took 2188 ms. Trace:
I20240503 14:12:17.825639 28661 rpcz_store.cc:276] 0503 14:12:15.637086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.637158 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:17.762177 (+2125019us) ts_tablet_manager.cc:1120] Deleting tablet 962df44408484e3f8d7b7138f27c1a66
0503 14:12:17.762191 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.787440 (+ 25249us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.790495 (+ 3055us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.825395 (+ 34900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2124607}
I20240503 14:12:17.826139 28661 tablet_replica.cc:330] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.826463 28661 raft_consensus.cc:2229] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.826732 28661 raft_consensus.cc:2258] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.827769 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2331aec871824a238f3f64de217d7092 (table default.Table_9d9d27627b064475a2f86b5ddcf4236f [id=2d1971a7d562426886d85395dfa1b299]) successfully deleted
I20240503 14:12:17.828624 28661 ts_tablet_manager.cc:1875] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:17.814563 28150 leader_election.cc:341] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.829697 28150 leader_election.cc:302] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.830478 29602 raft_consensus.cc:2726] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:17.831076 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 62e4f6e3908e420e9fa156ee24b437d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e4f6e3908e420e9fa156ee24b437d6 already in progress: creating tablet
W20240503 14:12:17.831516 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0f75e102dd4249c3800e1fcb4339acf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f75e102dd4249c3800e1fcb4339acf3 already in progress: creating tablet
W20240503 14:12:17.831827 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 1292a339f00247369ebeaa35752d51f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1292a339f00247369ebeaa35752d51f8 already in progress: creating tablet
I20240503 14:12:17.833770 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2404807f8500437ababca53f6388b636 (table default.Table_7d8f2c48beb5472fb05f0cbd475f7182 [id=a396cc6875ba41d29e801308d3f274d0]) successfully deleted
W20240503 14:12:17.834335 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f6061918660e4070a0c8339cb5009ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6061918660e4070a0c8339cb5009ff8 already in progress: creating tablet
I20240503 14:12:17.835366 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 140de4d395d54b7696062ed96b8aa89c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.836602 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.838724 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]) successfully deleted
W20240503 14:12:17.840919 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac577af6460b423288a790e532b00f82 consensus state has no leader
W20240503 14:12:17.841351 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7218cefb13ea4983b2111ed5fd0c997b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7218cefb13ea4983b2111ed5fd0c997b already in progress: creating tablet
W20240503 14:12:17.841727 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ffcfc89a415e4c68b6a54318755fe0d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffcfc89a415e4c68b6a54318755fe0d7 already in progress: creating tablet
I20240503 14:12:17.842126 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]) successfully deleted
W20240503 14:12:17.842304 28030 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 65) took 800 ms (client timeout 1000 ms). Trace:
W20240503 14:12:17.842435 28030 rpcz_store.cc:269] 0503 14:12:17.041892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.095619 (+ 53727us) service_pool.cc:225] Handling call
0503 14:12:17.842192 (+746573us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:17.406165 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:17.408584 (+ 2419us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:17.408594 (+ 10us) write_op.cc:435] Acquired schema lock
0503 14:12:17.408597 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:17.415928 (+ 7331us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:17.416252 (+ 324us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:17.416263 (+ 11us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:17.417537 (+ 1274us) tablet.cc:699] Row locks acquired
0503 14:12:17.417542 (+ 5us) write_op.cc:260] PREPARE: finished
0503 14:12:17.418292 (+ 750us) write_op.cc:270] Start()
0503 14:12:17.424477 (+ 6185us) write_op.cc:276] Timestamp: P: 1714745537417940 usec, L: 0
0503 14:12:17.424778 (+ 301us) op_driver.cc:331] REPLICATION: starting
0503 14:12:17.429393 (+ 4615us) log.cc:842] Serialized 533 byte log entry
0503 14:12:17.837916 (+408523us) op_driver.cc:444] REPLICATION: finished
0503 14:12:17.838191 (+ 275us) write_op.cc:301] APPLY: starting
0503 14:12:17.838208 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:17.838272 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:17.838310 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:17.838359 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:17.838390 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:17.838396 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:17.838466 (+ 70us) log.cc:842] Serialized 23 byte log entry
0503 14:12:17.838537 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:17.838555 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:17.838561 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:17.838621 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:17.838624 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":225,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":487,"mutex_wait_us":2693,"num_ops":1,"prepare.queue_time_us":122445,"prepare.run_cpu_time_us":1876,"prepare.run_wall_time_us":25316,"replication_time_us":413118,"spinlock_wait_cycles":14592}]]}
I20240503 14:12:17.840443 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5757e6e8703640519f7d38754cc0c8ce
I20240503 14:12:17.846837 28661 ts_tablet_manager.cc:1888] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.846812 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]). Expected version 3 got 0
I20240503 14:12:17.847107 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
W20240503 14:12:17.848273 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64bae128469f4e8bb2d743320731446a consensus state has no leader
I20240503 14:12:17.848817 28661 log.cc:1196] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/50edc38cdbe74397aa4f4928d56d9fbc
I20240503 14:12:17.849188 28661 ts_tablet_manager.cc:1909] T 50edc38cdbe74397aa4f4928d56d9fbc P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.850059 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 629) took 1870 ms. Trace:
I20240503 14:12:17.850158 28661 rpcz_store.cc:276] 0503 14:12:15.979439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:15.979517 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:17.826075 (+1846558us) ts_tablet_manager.cc:1120] Deleting tablet 50edc38cdbe74397aa4f4928d56d9fbc
0503 14:12:17.826088 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.836525 (+ 10437us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.846706 (+ 10181us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.850030 (+ 3324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846199}
I20240503 14:12:17.850606 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 630) took 1745 ms. Trace:
I20240503 14:12:17.851502 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_960c4947e6bc4bb6839d7257e7d93a41" } new_table_name: "default.Table_c520ee9887734e36b4407737247c1e76" modify_external_catalogs: true
I20240503 14:12:17.851917 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e45df464f6954a56bec8b107b350320c
I20240503 14:12:17.852857 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet e45df464f6954a56bec8b107b350320c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.853655 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 5757e6e8703640519f7d38754cc0c8ce deleted and replaced by tablet 2c1db04ab92c4af0847730a47635a697
I20240503 14:12:17.853811 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet e45df464f6954a56bec8b107b350320c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.850706 28661 rpcz_store.cc:276] 0503 14:12:16.104958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.105021 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:17.850445 (+1745424us) ts_tablet_manager.cc:1120] Deleting tablet c9325d5142af4b018354bf38370b61e0
0503 14:12:17.850457 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.850576 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745156}
I20240503 14:12:17.854419 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 631) took 1748 ms. Trace:
I20240503 14:12:17.854610 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757e6e8703640519f7d38754cc0c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet 2c1db04ab92c4af0847730a47635a697) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.854568 28661 rpcz_store.cc:276] 0503 14:12:16.105968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.106026 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:17.854250 (+1748224us) ts_tablet_manager.cc:1120] Deleting tablet 74a9d58fb30d4b60ae0a44ec217fea11
0503 14:12:17.854264 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.854394 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748007}
I20240503 14:12:17.854981 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 633) took 1745 ms. Trace:
I20240503 14:12:17.855139 28661 rpcz_store.cc:276] 0503 14:12:16.109876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.109933 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:17.854852 (+1744919us) ts_tablet_manager.cc:1120] Deleting tablet 5a081ba59ad44890bf2f94874c034277
0503 14:12:17.854862 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.854964 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1744475}
I20240503 14:12:17.855567 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 632) took 1746 ms. Trace:
I20240503 14:12:17.855717 28661 rpcz_store.cc:276] 0503 14:12:16.108876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.108927 (+ 51us) service_pool.cc:225] Handling call
0503 14:12:17.855448 (+1746521us) ts_tablet_manager.cc:1120] Deleting tablet 31ddd791d4e14630bfb3b4b1513f76e1
0503 14:12:17.855458 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.855549 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1744818}
I20240503 14:12:17.855784 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757e6e8703640519f7d38754cc0c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet 2c1db04ab92c4af0847730a47635a697) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.856160 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7aba6cf95a4e4459aa96ea157115e3d2" } modify_external_catalogs: true
I20240503 14:12:17.856415 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4db31dc8ab934c1595422943e50fe9c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:17.856779 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 634) took 1745 ms. Trace:
I20240503 14:12:17.857113 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_75f5b842e7054f3787e97d6641a08485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.857702 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75f5b842e7054f3787e97d6641a08485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.857854 28661 rpcz_store.cc:276] 0503 14:12:16.111115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.111204 (+ 89us) service_pool.cc:225] Handling call
0503 14:12:17.856502 (+1745298us) ts_tablet_manager.cc:1120] Deleting tablet aac51b83abe14ab5b1fb5d627233e58e
0503 14:12:17.856511 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.856619 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745108}
W20240503 14:12:17.801437 28427 leader_election.cc:341] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.859409 28427 leader_election.cc:302] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:17.860247 29652 raft_consensus.cc:2726] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:17.860340 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 58917.1us
I20240503 14:12:17.859635 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 635) took 1627 ms. Trace:
I20240503 14:12:17.807479 28604 tablet_bootstrap.cc:492] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.860886 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3d175bb978ee4ad5b87f72b8cb198e0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:17.861307 28604 ts_tablet_manager.cc:1367] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.154s user 0.004s sys 0.004s
W20240503 14:12:17.861572 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d175bb978ee4ad5b87f72b8cb198e0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.862099 28604 raft_consensus.cc:348] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.862303 28604 raft_consensus.cc:374] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.862355 28604 raft_consensus.cc:729] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.862711 28604 consensus_queue.cc:260] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.862941 29652 raft_consensus.cc:482] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.863039 29652 raft_consensus.cc:504] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.863559 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7aba6cf95a4e4459aa96ea157115e3d2" } modify_external_catalogs: true
I20240503 14:12:17.863687 29652 leader_election.cc:288] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:17.865705 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4db31dc8ab934c1595422943e50fe9c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.866250 28604 ts_tablet_manager.cc:1398] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:17.866672 28604 tablet_bootstrap.cc:492] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.868448 28604 tablet_bootstrap.cc:654] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.868605 28661 rpcz_store.cc:276] 0503 14:12:16.232018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.232701 (+ 683us) service_pool.cc:225] Handling call
0503 14:12:17.858234 (+1625533us) ts_tablet_manager.cc:1120] Deleting tablet a26e8389639748ae99832d7a7a9d5af3
0503 14:12:17.858245 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.859605 (+ 1360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1553951}
I20240503 14:12:17.870474 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7d57fc4486457e9848daad79f04356 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.870666 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757e6e8703640519f7d38754cc0c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet 2c1db04ab92c4af0847730a47635a697) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.872032 28661 tablet_replica.cc:330] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.872107 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3307e69a5d4f4765b6ffc39e9df66fa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:17.872278 29602 raft_consensus.cc:482] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.872499 29602 raft_consensus.cc:504] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:17.872856 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3307e69a5d4f4765b6ffc39e9df66fa1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.873345 29602 leader_election.cc:288] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:17.873601 28661 raft_consensus.cc:2229] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.873775 28661 raft_consensus.cc:2258] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.874330 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet e135b60660f448a7974520f2f87cf93a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.874405 28661 ts_tablet_manager.cc:1875] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.874596 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d4e4a1f0b14406be957426a45bf45d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.874763 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet e45df464f6954a56bec8b107b350320c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:17.876492 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:17.877465 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:17.878360 28150 leader_election.cc:341] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:17.878682 28150 leader_election.cc:341] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.878809 28150 leader_election.cc:302] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:17.879429 29602 raft_consensus.cc:2726] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:17.886058 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25c852aa7aa4fbfafdacec43f5bc426" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.886373 28352 raft_consensus.cc:2454] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:17.887123 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 962df44408484e3f8d7b7138f27c1a66 (table default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]) successfully deleted
I20240503 14:12:17.886714 28653 ts_tablet_manager.cc:1888] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.887517 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:17.888758 28653 log.cc:1196] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7218cefb13ea4983b2111ed5fd0c997b
I20240503 14:12:17.890136 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 50edc38cdbe74397aa4f4928d56d9fbc (table default.Table_9a954e776a184d819708a62a99ed94fc [id=713e339595f142ab8a82bb724d23a487]) successfully deleted
W20240503 14:12:17.890803 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c9325d5142af4b018354bf38370b61e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9325d5142af4b018354bf38370b61e0 already in progress: creating tablet
I20240503 14:12:17.891990 28653 ts_tablet_manager.cc:1909] T 7218cefb13ea4983b2111ed5fd0c997b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:17.892499 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 74a9d58fb30d4b60ae0a44ec217fea11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74a9d58fb30d4b60ae0a44ec217fea11 already in progress: creating tablet
W20240503 14:12:17.893023 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5a081ba59ad44890bf2f94874c034277 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a081ba59ad44890bf2f94874c034277 already in progress: creating tablet
W20240503 14:12:17.895184 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 31ddd791d4e14630bfb3b4b1513f76e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31ddd791d4e14630bfb3b4b1513f76e1 already in progress: creating tablet
W20240503 14:12:17.895596 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet aac51b83abe14ab5b1fb5d627233e58e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aac51b83abe14ab5b1fb5d627233e58e already in progress: creating tablet
I20240503 14:12:17.896664 28614 tablet_bootstrap.cc:492] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
W20240503 14:12:17.897116 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a26e8389639748ae99832d7a7a9d5af3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a26e8389639748ae99832d7a7a9d5af3 already in progress: creating tablet
W20240503 14:12:17.897610 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 11d9f47543524fc5921c99aad869035f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11d9f47543524fc5921c99aad869035f already in progress: creating tablet
I20240503 14:12:17.897732 29652 raft_consensus.cc:482] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.897933 28614 ts_tablet_manager.cc:1367] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.001s
I20240503 14:12:17.897917 29652 raft_consensus.cc:504] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.898558 29652 leader_election.cc:288] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:17.898841 28614 raft_consensus.cc:348] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:17.899425 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e21a5dafa82f4660927d96c779a23478 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e21a5dafa82f4660927d96c779a23478 already in progress: creating tablet
W20240503 14:12:17.900017 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 16fed718124a4640967ab13a1ac2b379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16fed718124a4640967ab13a1ac2b379 already in progress: creating tablet
W20240503 14:12:17.900866 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 007c84fd9d9247728b0babd48689aba8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 007c84fd9d9247728b0babd48689aba8 already in progress: creating tablet
I20240503 14:12:17.900846 28653 tablet_replica.cc:330] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:17.902772 28653 raft_consensus.cc:2229] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:17.904652 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e45df464f6954a56bec8b107b350320c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45df464f6954a56bec8b107b350320c already in progress: creating tablet
I20240503 14:12:17.904999 28614 raft_consensus.cc:374] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.905309 28614 raft_consensus.cc:729] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.904470 28661 ts_tablet_manager.cc:1888] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:17.905757 28614 consensus_queue.cc:260] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.905954 28661 log.cc:1196] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/004d556f3ed44a7db1a52d205ecfd90b
I20240503 14:12:17.904606 28427 leader_election.cc:302] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:17.906126 28653 pending_rounds.cc:62] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
W20240503 14:12:17.906523 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5757e6e8703640519f7d38754cc0c8ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5757e6e8703640519f7d38754cc0c8ce already in progress: creating tablet
I20240503 14:12:17.906805 29701 raft_consensus.cc:2781] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:17.906970 29701 raft_consensus.cc:482] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:17.907162 29701 raft_consensus.cc:3037] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.904691 29700 consensus_queue.cc:1035] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:12:17.906960 28653 pending_rounds.cc:68] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597722646351872 op_type: NO_OP noop_request { }
I20240503 14:12:17.906363 28661 ts_tablet_manager.cc:1909] T 004d556f3ed44a7db1a52d205ecfd90b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:17.908824 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 637) took 1675 ms. Trace:
I20240503 14:12:17.908949 28661 rpcz_store.cc:276] 0503 14:12:16.233197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.233334 (+ 137us) service_pool.cc:225] Handling call
0503 14:12:17.871969 (+1638635us) ts_tablet_manager.cc:1120] Deleting tablet 004d556f3ed44a7db1a52d205ecfd90b
0503 14:12:17.871980 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:17.886472 (+ 14492us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.904311 (+ 17839us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:17.908795 (+ 4484us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566888}
I20240503 14:12:17.909330 28661 tablet_replica.cc:330] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:17.908625 28653 raft_consensus.cc:2866] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:17.909536 28661 raft_consensus.cc:2229] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:17.910234 29701 raft_consensus.cc:504] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.910020 28653 raft_consensus.cc:2258] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.914991 28600 tablet_bootstrap.cc:492] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.915227 28600 ts_tablet_manager.cc:1367] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.106s user 0.006s sys 0.006s
I20240503 14:12:17.916373 28600 raft_consensus.cc:348] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.916872 28600 raft_consensus.cc:374] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.917014 28600 raft_consensus.cc:729] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.917570 28600 consensus_queue.cc:260] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.918292 28614 ts_tablet_manager.cc:1398] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240503 14:12:17.918749 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7218cefb13ea4983b2111ed5fd0c997b (table default.Table_e3c825867cf84b72bfb708fcc39f6acb [id=3a24797cc53b439da44cc77025fa9f92]) successfully deleted
I20240503 14:12:17.911190 28604 tablet_bootstrap.cc:492] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.919464 28600 ts_tablet_manager.cc:1398] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:17.919464 28604 ts_tablet_manager.cc:1367] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.000s
I20240503 14:12:17.920421 28604 raft_consensus.cc:348] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.920753 28604 raft_consensus.cc:374] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.920871 28604 raft_consensus.cc:729] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.921114 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:17.921015 28653 ts_tablet_manager.cc:1875] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.921154 28604 consensus_queue.cc:260] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.921953 28604 ts_tablet_manager.cc:1398] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:17.922344 28604 tablet_bootstrap.cc:492] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.923933 28604 tablet_bootstrap.cc:654] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.925815 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf5d153ab5142c0902cc6e96629f412" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.934260 28604 tablet_bootstrap.cc:492] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.934552 28604 ts_tablet_manager.cc:1367] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240503 14:12:17.935449 28604 raft_consensus.cc:348] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.935875 28604 raft_consensus.cc:374] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.936007 28604 raft_consensus.cc:729] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.936376 28604 consensus_queue.cc:260] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.937191 28600 tablet_bootstrap.cc:492] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:17.935667 28614 tablet_bootstrap.cc:492] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:17.938387 28604 ts_tablet_manager.cc:1398] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:17.938930 28604 tablet_bootstrap.cc:492] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.940528 28600 tablet_bootstrap.cc:654] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.940730 28604 tablet_bootstrap.cc:654] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.943121 28614 tablet_bootstrap.cc:654] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:17.943328 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3023785b11c4ad7b8757a715f0952c5 consensus state has no leader
I20240503 14:12:17.943838 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_602dce62ca0141aeae93bfbc8722054d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.945127 28033 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 87) took 905 ms (client timeout 1000 ms). Trace:
I20240503 14:12:17.945520 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 with delete_type TABLET_DATA_DELETED (replaced by tablet d4c8a97076a44f9eaa3a15026f90a304) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:17.945286 28033 rpcz_store.cc:269] 0503 14:12:17.039404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.092776 (+ 53372us) service_pool.cc:225] Handling call
0503 14:12:17.945069 (+852293us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:17.299169 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:17.302727 (+ 3558us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:17.302734 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:17.302737 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:17.326961 (+ 24224us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:17.326976 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:17.327170 (+ 194us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:17.328416 (+ 1246us) tablet.cc:699] Row locks acquired
0503 14:12:17.328420 (+ 4us) write_op.cc:260] PREPARE: finished
0503 14:12:17.328842 (+ 422us) write_op.cc:270] Start()
0503 14:12:17.329248 (+ 406us) write_op.cc:276] Timestamp: P: 1714745537328632 usec, L: 0
0503 14:12:17.329255 (+ 7us) op_driver.cc:331] REPLICATION: starting
0503 14:12:17.339521 (+ 10266us) log.cc:842] Serialized 356 byte log entry
0503 14:12:17.476484 (+136963us) op_driver.cc:444] REPLICATION: finished
0503 14:12:17.476653 (+ 169us) write_op.cc:301] APPLY: starting
0503 14:12:17.476677 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:17.476742 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:17.476782 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:17.476856 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:17.476901 (+ 45us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:17.476911 (+ 10us) write_op.cc:312] APPLY: finished
0503 14:12:17.477020 (+ 109us) log.cc:842] Serialized 23 byte log entry
0503 14:12:17.477113 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:17.477137 (+ 24us) write_op.cc:454] Released schema lock
0503 14:12:17.477146 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:17.477228 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:17.477232 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1925,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":72,"apply.run_cpu_time_us":659,"apply.run_wall_time_us":659,"mutex_wait_us":6883,"num_ops":1,"prepare.queue_time_us":19279,"prepare.run_cpu_time_us":1757,"prepare.run_wall_time_us":87487,"replication_time_us":147192}]]}
I20240503 14:12:17.946231 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 1603e2360041427f8635782e812b6431 with delete_type TABLET_DATA_DELETED (replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:17.946213 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ff5fa02257a5423785ae99a66c759e9c" } new_table_name: "default.Table_99b171729b6f4232a36bb9ba33008749" modify_external_catalogs: true
I20240503 14:12:17.948019 28603 tablet_replica.cc:330] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:17.948273 28603 raft_consensus.cc:2229] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:17.950178 28603 pending_rounds.cc:62] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:17.950841 28603 pending_rounds.cc:68] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597722658484224 op_type: NO_OP noop_request { }
I20240503 14:12:17.951472 28603 raft_consensus.cc:2866] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:17.951962 28603 raft_consensus.cc:2258] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:17.952591 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_602dce62ca0141aeae93bfbc8722054d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.953476 28603 ts_tablet_manager.cc:1875] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.958035 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_54d6cac450a145a89afabfe0c688c70c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.958772 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54d6cac450a145a89afabfe0c688c70c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:17.960312 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4639ff7138f4c3ea6b7e5e5b4e7cd6d consensus state has no leader
I20240503 14:12:17.960948 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f6d907b2cd874cd98d61854bb4c623fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:17.961573 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6d907b2cd874cd98d61854bb4c623fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:17.963177 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b8a25247ce2a4633b8e8bc77291fb69d" } modify_external_catalogs: true
I20240503 14:12:17.963397 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b8a25247ce2a4633b8e8bc77291fb69d" } modify_external_catalogs: true
I20240503 14:12:17.964298 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3" } modify_external_catalogs: true
I20240503 14:12:17.964540 28604 tablet_bootstrap.cc:492] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:17.964526 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3" } modify_external_catalogs: true
I20240503 14:12:17.964718 28604 ts_tablet_manager.cc:1367] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 14:12:17.965662 28604 raft_consensus.cc:348] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.965958 28604 raft_consensus.cc:374] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.966146 28604 raft_consensus.cc:729] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.966491 28604 consensus_queue.cc:260] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.967267 28604 ts_tablet_manager.cc:1398] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:17.969980 28614 tablet_bootstrap.cc:492] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:17.970222 28614 ts_tablet_manager.cc:1367] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.002s
I20240503 14:12:17.971210 28614 raft_consensus.cc:348] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.971531 28614 raft_consensus.cc:374] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.971760 28614 raft_consensus.cc:729] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.972103 28614 consensus_queue.cc:260] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.973006 28604 tablet_bootstrap.cc:492] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:17.974848 29700 raft_consensus.cc:482] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.975010 29700 raft_consensus.cc:504] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.975754 29700 leader_election.cc:288] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:17.924415 28661 raft_consensus.cc:2258] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:17.924630 29701 leader_election.cc:288] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:17.976915 28427 leader_election.cc:341] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b25c852aa7aa4fbfafdacec43f5bc426
I20240503 14:12:17.978763 28614 ts_tablet_manager.cc:1398] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20240503 14:12:17.979375 28427 leader_election.cc:341] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: daf5d153ab5142c0902cc6e96629f412
I20240503 14:12:17.978286 28604 tablet_bootstrap.cc:654] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.981135 28661 ts_tablet_manager.cc:1875] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:17.983095 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25c852aa7aa4fbfafdacec43f5bc426" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:17.983363 28353 raft_consensus.cc:3037] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.985606 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 004d556f3ed44a7db1a52d205ecfd90b (table default.Table_5264956c580441a6a23c96d39a41ea71 [id=59887894cf0843df9cd01850035c1e64]) successfully deleted
I20240503 14:12:17.986196 28353 raft_consensus.cc:2454] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:17.990434 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fc189ad8c19041f7814fac6de9f482cb" } modify_external_catalogs: true
I20240503 14:12:17.990710 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fc189ad8c19041f7814fac6de9f482cb" } modify_external_catalogs: true
W20240503 14:12:17.992053 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9c7b5fb9b4a483c8292f68387591034 consensus state has no leader
I20240503 14:12:17.992376 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet cf810e84367f420b928a171790849445 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:17.992774 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c924a74ee836406fb3157baef6f4c901" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:17.992884 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c924a74ee836406fb3157baef6f4c901" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:17.993047 28214 raft_consensus.cc:2454] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:17.993103 28353 raft_consensus.cc:2454] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:17.993562 29700 raft_consensus.cc:482] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:17.993712 29700 raft_consensus.cc:504] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:17.994427 29700 leader_election.cc:288] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:17.996589 28427 consensus_peers.cc:485] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a32b6995818041778617a0d6bf936c8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:17.997819 28600 tablet_bootstrap.cc:492] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:17.998715 28600 ts_tablet_manager.cc:1367] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.004s
W20240503 14:12:17.999380 28427 leader_election.cc:341] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:17.999590 28427 leader_election.cc:302] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.000479 29700 raft_consensus.cc:2726] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:18.000494 27856 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 72574.7us
I20240503 14:12:18.000025 28600 raft_consensus.cc:348] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.001549 28427 leader_election.cc:302] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.001868 28600 raft_consensus.cc:374] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.002173 28600 raft_consensus.cc:729] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.002596 29700 raft_consensus.cc:2781] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.002723 29700 raft_consensus.cc:686] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:18.002974 29700 consensus_queue.cc:237] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.003161 28604 tablet_bootstrap.cc:492] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.003314 28604 ts_tablet_manager.cc:1367] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240503 14:12:18.004160 28604 raft_consensus.cc:348] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.004405 28604 raft_consensus.cc:374] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.004489 28604 raft_consensus.cc:729] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.004628 28600 consensus_queue.cc:260] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.004751 28604 consensus_queue.cc:260] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.005641 28604 ts_tablet_manager.cc:1398] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:18.006014 28604 tablet_bootstrap.cc:492] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.008476 28604 tablet_bootstrap.cc:654] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.009264 29456 raft_consensus.cc:482] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.009428 29456 raft_consensus.cc:504] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.010203 29456 leader_election.cc:288] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:18.011476 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.011709 28489 raft_consensus.cc:2454] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:18.012329 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:18.012514 28653 ts_tablet_manager.cc:1888] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:18.012687 28290 leader_election.cc:302] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.013129 29456 raft_consensus.cc:2781] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:18.013216 29456 raft_consensus.cc:482] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:18.013278 29456 raft_consensus.cc:3037] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.008940 28427 leader_election.cc:302] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.016273 29456 raft_consensus.cc:504] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.012655 28214 raft_consensus.cc:2454] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:18.016683 28653 log.cc:1196] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/65cdd090bf0f493f8b92341369378532
I20240503 14:12:18.017158 29456 leader_election.cc:288] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:18.017411 28653 ts_tablet_manager.cc:1909] T 65cdd090bf0f493f8b92341369378532 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:18.018085 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:18.018464 28214 raft_consensus.cc:3037] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.021519 28214 raft_consensus.cc:2454] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:18.022619 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet ff1958dc488e4854a8c8e74da9fab243 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:18.023066 28290 leader_election.cc:302] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:18.023249 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 9aaf7d4927674eedbfd486df6a347f7c from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:18.023999 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 65cdd090bf0f493f8b92341369378532 (table default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c [id=d16dae55faab45f4bb6337e042735ad0]) successfully deleted
I20240503 14:12:18.024343 29701 raft_consensus.cc:2781] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:18.024443 29701 raft_consensus.cc:482] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:18.024505 29701 raft_consensus.cc:3037] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.026636 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:18.026858 28489 raft_consensus.cc:3037] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.027161 29701 raft_consensus.cc:504] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.028584 29701 leader_election.cc:288] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.029311 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.029508 28489 raft_consensus.cc:2454] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:18.029574 28353 raft_consensus.cc:2379] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:18.030015 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0795253db548fb84c2d7bb3dc48335" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:18.030138 29456 raft_consensus.cc:2781] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.030277 29456 raft_consensus.cc:686] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:18.030501 28214 raft_consensus.cc:2379] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:18.030541 29456 consensus_queue.cc:237] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.031567 28604 tablet_bootstrap.cc:492] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.031780 28604 ts_tablet_manager.cc:1367] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
W20240503 14:12:18.032469 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ffcfc89a415e4c68b6a54318755fe0d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffcfc89a415e4c68b6a54318755fe0d7 already in progress: creating tablet
I20240503 14:12:18.033213 28604 raft_consensus.cc:348] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.033571 28604 raft_consensus.cc:374] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.033679 28604 raft_consensus.cc:729] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.033988 28604 consensus_queue.cc:260] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.034900 28604 ts_tablet_manager.cc:1398] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:18.035272 28604 tablet_bootstrap.cc:492] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.040963 28604 tablet_bootstrap.cc:654] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.041549 28600 ts_tablet_manager.cc:1398] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240503 14:12:18.041741 28614 tablet_bootstrap.cc:492] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:18.043838 28614 tablet_bootstrap.cc:654] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.044507 28653 tablet_replica.cc:330] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:18.045862 28653 raft_consensus.cc:2229] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:18.046370 28427 leader_election.cc:341] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b25c852aa7aa4fbfafdacec43f5bc426
I20240503 14:12:18.047078 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_eb6121fb3aa742a3a8aa70b1426d9777"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.047783 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb6121fb3aa742a3a8aa70b1426d9777 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.045356 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c924a74ee836406fb3157baef6f4c901" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:18.049458 28214 raft_consensus.cc:3037] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.051496 28600 tablet_bootstrap.cc:492] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:18.047701 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:18.059334 28604 tablet_bootstrap.cc:492] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.059628 28604 ts_tablet_manager.cc:1367] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240503 14:12:18.061852 28024 catalog_manager.cc:5484] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:18.062433 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 0e680f189add4b5c96de4046321aad2d (table default.Table_2d7ecf8640f64d82908807b4c4d4fd7f [id=d1088a94cd0442c9bd9837bf9f0e1ff9]). Expected version 2 got 0
I20240503 14:12:18.062968 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:18.063505 28604 raft_consensus.cc:348] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.063787 28604 raft_consensus.cc:374] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.063903 28604 raft_consensus.cc:729] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.064246 28604 consensus_queue.cc:260] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.065017 28604 ts_tablet_manager.cc:1398] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:18.065410 28604 tablet_bootstrap.cc:492] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.061398 28427 leader_election.cc:302] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.066486 29701 raft_consensus.cc:2726] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.069589 28604 tablet_bootstrap.cc:654] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.052649 28653 raft_consensus.cc:2258] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.065055 28661 ts_tablet_manager.cc:1888] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.083029 28661 log.cc:1196] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ca6627f0d9a544bb8176bea6c1af5ab4
I20240503 14:12:18.045300 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c924a74ee836406fb3157baef6f4c901" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:18.083925 28353 raft_consensus.cc:3037] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.086369 28653 ts_tablet_manager.cc:1875] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.087080 28353 raft_consensus.cc:2454] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:18.082978 28603 ts_tablet_manager.cc:1888] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:18.088330 28603 log.cc:1196] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/8b1728b6f0e64661aaa8184ce51366d2
I20240503 14:12:18.088359 28427 leader_election.cc:302] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.088742 29701 raft_consensus.cc:2781] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.088811 28603 ts_tablet_manager.cc:1909] T 8b1728b6f0e64661aaa8184ce51366d2 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:18.088879 29701 raft_consensus.cc:686] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:18.088860 28600 tablet_bootstrap.cc:654] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.089135 29701 consensus_queue.cc:237] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.090344 28603 tablet_replica.cc:330] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:18.090543 28603 raft_consensus.cc:2229] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.090693 28603 raft_consensus.cc:2258] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.091289 28603 ts_tablet_manager.cc:1875] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.096717 28661 ts_tablet_manager.cc:1909] T ca6627f0d9a544bb8176bea6c1af5ab4 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:18.097294 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 638) took 1863 ms. Trace:
I20240503 14:12:18.097476 28661 rpcz_store.cc:276] 0503 14:12:16.233323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.233534 (+ 211us) service_pool.cc:225] Handling call
0503 14:12:17.909273 (+1675739us) ts_tablet_manager.cc:1120] Deleting tablet ca6627f0d9a544bb8176bea6c1af5ab4
0503 14:12:17.909284 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.009512 (+100228us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.064936 (+ 55424us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.097262 (+ 32326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1604123}
I20240503 14:12:18.098048 28661 tablet_replica.cc:330] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:18.098362 28661 raft_consensus.cc:2229] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.098625 28661 raft_consensus.cc:2258] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.099433 28661 ts_tablet_manager.cc:1875] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.099704 28604 tablet_bootstrap.cc:492] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.099927 28604 ts_tablet_manager.cc:1367] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20240503 14:12:18.101039 28604 raft_consensus.cc:348] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.101382 28604 raft_consensus.cc:374] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.101490 28604 raft_consensus.cc:729] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.101820 28604 consensus_queue.cc:260] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.102608 28604 ts_tablet_manager.cc:1398] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:18.102990 28604 tablet_bootstrap.cc:492] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.104298 28214 raft_consensus.cc:2454] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:18.108332 28604 tablet_bootstrap.cc:654] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.109016 28600 tablet_bootstrap.cc:492] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:18.109421 28600 ts_tablet_manager.cc:1367] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.002s
I20240503 14:12:18.110925 28600 raft_consensus.cc:348] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.111516 28600 raft_consensus.cc:374] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.111712 28600 raft_consensus.cc:729] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.112336 28600 consensus_queue.cc:260] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.114897 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ca6627f0d9a544bb8176bea6c1af5ab4 (table default.Table_e3e1acbcbb7c4aeab3db92cc253b441d [id=9ac543611bee46c08fb673c287aa6af8]) successfully deleted
I20240503 14:12:18.115242 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 8b1728b6f0e64661aaa8184ce51366d2 (table default.Table_80d78826082349c882ea2c0ca66e0141 [id=bc70f87f0df24f569c3cc1c65a86f667]) successfully deleted
I20240503 14:12:18.117429 28600 ts_tablet_manager.cc:1398] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:12:18.125100 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 215a6f53e47e49cb984565adf659614d
I20240503 14:12:18.126070 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3d175bb978ee4ad5b87f72b8cb198e0d" } new_table_name: "default.Table_caa773d21464476e8a52534f15c92525" modify_external_catalogs: true
I20240503 14:12:18.127868 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5cdc868ddf0a490292eae78ebc8c8ec6
I20240503 14:12:18.128257 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 215a6f53e47e49cb984565adf659614d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.128499 28603 ts_tablet_manager.cc:1888] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.128990 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4a597910101d4133a1ad5201ce76c839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:18.129211 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 7b89711872ba442ca66e35091a2271da (DEFAULT_TABLE table=default.Table_7c688a2e434147e582bcf88eeafc227b [id=158a51a748c94f6d9f2e6eeb100ecb1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.129540 28604 tablet_bootstrap.cc:492] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.129688 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b89711872ba442ca66e35091a2271da. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:18.129686 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a597910101d4133a1ad5201ce76c839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.129794 28604 ts_tablet_manager.cc:1367] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.002s
I20240503 14:12:18.130770 28604 raft_consensus.cc:348] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.130982 28604 raft_consensus.cc:374] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.131042 28604 raft_consensus.cc:729] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.131278 28604 consensus_queue.cc:260] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.131798 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cafee16021144efc9522addcc165ad89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:18.132159 28604 ts_tablet_manager.cc:1398] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:18.132412 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 833ad747feca411aa480dd420a6c02cc
I20240503 14:12:18.132483 28604 tablet_bootstrap.cc:492] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.128616 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 7b89711872ba442ca66e35091a2271da (DEFAULT_TABLE table=default.Table_7c688a2e434147e582bcf88eeafc227b [id=158a51a748c94f6d9f2e6eeb100ecb1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.128701 28603 log.cc:1196] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/1603e2360041427f8635782e812b6431
I20240503 14:12:18.133316 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a28d952dbd9141a9a113c42bcbc27a14" } new_table_name: "default.Table_53c1790884a24325abc8160a18f4d9cf" modify_external_catalogs: true
I20240503 14:12:18.133631 28603 ts_tablet_manager.cc:1909] T 1603e2360041427f8635782e812b6431 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:18.134975 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9aaf7d4927674eedbfd486df6a347f7c
I20240503 14:12:18.135494 28025 catalog_manager.cc:6554] ReplaceTablet: tablet 9aaf7d4927674eedbfd486df6a347f7c deleted and replaced by tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad
I20240503 14:12:18.136927 28604 tablet_bootstrap.cc:654] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.137141 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 215a6f53e47e49cb984565adf659614d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.137465 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 215a6f53e47e49cb984565adf659614d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.137712 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.136996 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.137146 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 833ad747feca411aa480dd420a6c02cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.138974 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 7b89711872ba442ca66e35091a2271da (DEFAULT_TABLE table=default.Table_7c688a2e434147e582bcf88eeafc227b [id=158a51a748c94f6d9f2e6eeb100ecb1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.139355 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 833ad747feca411aa480dd420a6c02cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.139524 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b89711872ba442ca66e35091a2271da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.140874 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.141026 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 833ad747feca411aa480dd420a6c02cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.143447 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ec78ac6bb77c4e4790e3d2c0f107c6e7" } modify_external_catalogs: true
I20240503 14:12:18.143710 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ec78ac6bb77c4e4790e3d2c0f107c6e7" } modify_external_catalogs: true
I20240503 14:12:18.144238 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b89711872ba442ca66e35091a2271da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.150733 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 9db657eef2db4e3791318e01e1a965fb (DEFAULT_TABLE table=default.Table_39c390a1cebb4679b78b0e36750364d6 [id=ad0c9a77be164a91965c416b470e7a46]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:18.150884 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cafee16021144efc9522addcc165ad89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.151326 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db657eef2db4e3791318e01e1a965fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.152005 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 888c28447db64751adca55ae769d0a37 (DEFAULT_TABLE table=default.Table_f8c8d1a54b104cbb88a7030052a89be3 [id=bcaf1e8f01fc4906bf319ea69df40fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.152483 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888c28447db64751adca55ae769d0a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.156440 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 9db657eef2db4e3791318e01e1a965fb (DEFAULT_TABLE table=default.Table_39c390a1cebb4679b78b0e36750364d6 [id=ad0c9a77be164a91965c416b470e7a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.156487 28194 tablet_service.cc:1467] Processing CreateTablet for tablet f6adeb488fd34301915f2977ce8cf61d (DEFAULT_TABLE table=default.Table_091731e3847544b1b411f7fe14d3182a [id=7de9d225735e4e20ba4865caaae53bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.156874 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db657eef2db4e3791318e01e1a965fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.156916 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6adeb488fd34301915f2977ce8cf61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.160251 28600 tablet_bootstrap.cc:492] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:18.161213 28330 tablet_service.cc:1467] Processing CreateTablet for tablet f6adeb488fd34301915f2977ce8cf61d (DEFAULT_TABLE table=default.Table_091731e3847544b1b411f7fe14d3182a [id=7de9d225735e4e20ba4865caaae53bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.161201 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 888c28447db64751adca55ae769d0a37 (DEFAULT_TABLE table=default.Table_f8c8d1a54b104cbb88a7030052a89be3 [id=bcaf1e8f01fc4906bf319ea69df40fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.150738 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 888c28447db64751adca55ae769d0a37 (DEFAULT_TABLE table=default.Table_f8c8d1a54b104cbb88a7030052a89be3 [id=bcaf1e8f01fc4906bf319ea69df40fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.161201 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 9db657eef2db4e3791318e01e1a965fb (DEFAULT_TABLE table=default.Table_39c390a1cebb4679b78b0e36750364d6 [id=ad0c9a77be164a91965c416b470e7a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.165962 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888c28447db64751adca55ae769d0a37. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:18.170264 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet ac577af6460b423288a790e532b00f82 (table default.Table_7dc512798f934fd4a2d847c3f9ae1c16 [id=88d060012eef400b97c4a62d199d194a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ac577af6460b423288a790e532b00f82 consensus state has no leader
I20240503 14:12:18.173862 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 93bba058a8f147a1ae336b3be60a20b6 (DEFAULT_TABLE table=default.Table_749e46738ab841d4b031fea3fb61e96b [id=d59652e5536c4d629d9a174a0d9a2682]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.173862 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 93bba058a8f147a1ae336b3be60a20b6 (DEFAULT_TABLE table=default.Table_749e46738ab841d4b031fea3fb61e96b [id=d59652e5536c4d629d9a174a0d9a2682]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.174242 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 1603e2360041427f8635782e812b6431 (table default.Table_cf24dd3520bd4373acfd719f2c7737ad [id=e0a3f887fca8477ba19c4ed3738b1955]) successfully deleted
I20240503 14:12:18.174364 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bba058a8f147a1ae336b3be60a20b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.174402 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bba058a8f147a1ae336b3be60a20b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:18.174598 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 215a6f53e47e49cb984565adf659614d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215a6f53e47e49cb984565adf659614d already in progress: creating tablet
W20240503 14:12:18.175717 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cdc868ddf0a490292eae78ebc8c8ec6 already in progress: creating tablet
W20240503 14:12:18.176074 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 833ad747feca411aa480dd420a6c02cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 833ad747feca411aa480dd420a6c02cc already in progress: creating tablet
I20240503 14:12:18.176263 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aaf7d4927674eedbfd486df6a347f7c with delete_type TABLET_DATA_DELETED (replaced by tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.176755 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 93bba058a8f147a1ae336b3be60a20b6 (DEFAULT_TABLE table=default.Table_749e46738ab841d4b031fea3fb61e96b [id=d59652e5536c4d629d9a174a0d9a2682]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.177619 28324 tablet_service.cc:1467] Processing CreateTablet for tablet adc4677a01f14f7c87ca00b29c6238c6 (DEFAULT_TABLE table=default.Table_2aadd33c71aa4953a09c3d7a747e65da [id=aae577d3718e42638cf862fcb3f9adcc]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:18.192749 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9aaf7d4927674eedbfd486df6a347f7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aaf7d4927674eedbfd486df6a347f7c already in progress: creating tablet
I20240503 14:12:18.194413 28179 tablet_service.cc:1467] Processing CreateTablet for tablet adc4677a01f14f7c87ca00b29c6238c6 (DEFAULT_TABLE table=default.Table_2aadd33c71aa4953a09c3d7a747e65da [id=aae577d3718e42638cf862fcb3f9adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.195037 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc4677a01f14f7c87ca00b29c6238c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.196780 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aaf7d4927674eedbfd486df6a347f7c with delete_type TABLET_DATA_DELETED (replaced by tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.197577 28328 tablet_service.cc:1467] Processing CreateTablet for tablet f0a278bb9fe04a1a93c15da6cf38d27e (DEFAULT_TABLE table=default.Table_5f8d6960d51e4a63a42f434099952825 [id=3de052db21d849a897d812c18a4acd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.198638 28461 tablet_service.cc:1467] Processing CreateTablet for tablet f6adeb488fd34301915f2977ce8cf61d (DEFAULT_TABLE table=default.Table_091731e3847544b1b411f7fe14d3182a [id=7de9d225735e4e20ba4865caaae53bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.198411 28029 catalog_manager.cc:5484] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:18.199076 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:18.199330 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6adeb488fd34301915f2977ce8cf61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.201359 28189 tablet_service.cc:1467] Processing CreateTablet for tablet f0a278bb9fe04a1a93c15da6cf38d27e (DEFAULT_TABLE table=default.Table_5f8d6960d51e4a63a42f434099952825 [id=3de052db21d849a897d812c18a4acd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.201926 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a278bb9fe04a1a93c15da6cf38d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.202018 28029 catalog_manager.cc:5484] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:18.203047 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 37364d5facc241f4bbccc979a50a0ca8 (DEFAULT_TABLE table=default.Table_e4311f501e744a4a859e4eef2878127c [id=34f401c86fd44dd19d2deb03d65676b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.202702 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a3023785b11c4ad7b8757a715f0952c5 (table default.Table_facfcbde0b2740359615d4907998c7fa [id=750bb7e271c241d2bbf14bd91dbc42d0]). Expected version 1 got 0
I20240503 14:12:18.203612 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 37364d5facc241f4bbccc979a50a0ca8 (DEFAULT_TABLE table=default.Table_e4311f501e744a4a859e4eef2878127c [id=34f401c86fd44dd19d2deb03d65676b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.204013 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37364d5facc241f4bbccc979a50a0ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.203986 28329 tablet_service.cc:1467] Processing CreateTablet for tablet cc004243d5a947e3a8852605b0414c95 (DEFAULT_TABLE table=default.Table_3c5bac1089fd4f629005b1381e2bca47 [id=2a8294327c7c4c449620d245a8d2617a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.203990 28029 catalog_manager.cc:5484] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:18.204586 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:18.204867 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 37364d5facc241f4bbccc979a50a0ca8 (DEFAULT_TABLE table=default.Table_e4311f501e744a4a859e4eef2878127c [id=34f401c86fd44dd19d2deb03d65676b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.204898 28458 tablet_service.cc:1467] Processing CreateTablet for tablet f0a278bb9fe04a1a93c15da6cf38d27e (DEFAULT_TABLE table=default.Table_5f8d6960d51e4a63a42f434099952825 [id=3de052db21d849a897d812c18a4acd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.205322 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37364d5facc241f4bbccc979a50a0ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.205642 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a278bb9fe04a1a93c15da6cf38d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.205646 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 02dd4c914a4846bab3a00408247ec64a (table default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc]). Expected version 2 got 0
I20240503 14:12:18.205893 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:18.209064 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_83c9d941dac54f2b9d9b8798a7f56437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.209774 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83c9d941dac54f2b9d9b8798a7f56437 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.211099 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4e13e0a790bc495f93e8cabc45270a80" } new_table_name: "default.Table_892ecc42cdb14c05ba3b9d211ee5b9ae" modify_external_catalogs: true
I20240503 14:12:18.213479 28469 tablet_service.cc:1467] Processing CreateTablet for tablet adc4677a01f14f7c87ca00b29c6238c6 (DEFAULT_TABLE table=default.Table_2aadd33c71aa4953a09c3d7a747e65da [id=aae577d3718e42638cf862fcb3f9adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.214565 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc4677a01f14f7c87ca00b29c6238c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.219904 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:18.223044 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 with delete_type TABLET_DATA_DELETED (replaced by tablet d4c8a97076a44f9eaa3a15026f90a304) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.223528 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 1603e2360041427f8635782e812b6431 with delete_type TABLET_DATA_DELETED (replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.223874 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fb79fcf30d485798c64c815e372dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.224304 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2404807f8500437ababca53f6388b636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.224669 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cdd090bf0f493f8b92341369378532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.225019 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet daf5d153ab5142c0902cc6e96629f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.225400 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d9985eecda4f3194c3251216aa58ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.226744 28472 tablet_service.cc:1467] Processing CreateTablet for tablet cc004243d5a947e3a8852605b0414c95 (DEFAULT_TABLE table=default.Table_3c5bac1089fd4f629005b1381e2bca47 [id=2a8294327c7c4c449620d245a8d2617a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.228742 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d45788ae644a8581acca54cc55dd3f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.229099 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cdd090bf0f493f8b92341369378532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.229337 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 with delete_type TABLET_DATA_DELETED (replaced by tablet 70f0e07d32084999a3ff42f69bb1ca2f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.229578 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet c5297664a4d943c28adfb6ac4f061680 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.230083 28192 tablet_service.cc:1467] Processing CreateTablet for tablet cc004243d5a947e3a8852605b0414c95 (DEFAULT_TABLE table=default.Table_3c5bac1089fd4f629005b1381e2bca47 [id=2a8294327c7c4c449620d245a8d2617a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.230419 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc004243d5a947e3a8852605b0414c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.230602 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc004243d5a947e3a8852605b0414c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.232247 29700 raft_consensus.cc:482] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.232419 29700 raft_consensus.cc:504] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.232854 29701 raft_consensus.cc:482] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.233121 29700 leader_election.cc:288] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:18.233827 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7218cefb13ea4983b2111ed5fd0c997b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:18.234596 28427 leader_election.cc:341] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.234146 29701 raft_consensus.cc:504] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.236989 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:18.237460 28427 leader_election.cc:341] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 7218cefb13ea4983b2111ed5fd0c997b
I20240503 14:12:18.237638 28427 leader_election.cc:302] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.237780 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3023785b11c4ad7b8757a715f0952c5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:18.237965 29700 raft_consensus.cc:2726] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.238265 28214 raft_consensus.cc:2379] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:18.238840 28427 leader_election.cc:302] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.239182 29700 raft_consensus.cc:3037] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.241773 29700 raft_consensus.cc:2726] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.255010 29701 leader_election.cc:288] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.173887 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aaf7d4927674eedbfd486df6a347f7c with delete_type TABLET_DATA_DELETED (replaced by tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:18.266080 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a27d21de6a6441d810f379bba88f9a2 consensus state has no leader
W20240503 14:12:18.266916 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68c6bb1ce651489c9fcb6b839832d432 has no consensus state
I20240503 14:12:18.267529 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet cf810e84367f420b928a171790849445
I20240503 14:12:18.268186 28033 catalog_manager.cc:6554] ReplaceTablet: tablet cf810e84367f420b928a171790849445 deleted and replaced by tablet f154dfaeb4fe47238ad928c8fc3521da
I20240503 14:12:18.269418 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_602dce62ca0141aeae93bfbc8722054d" } new_table_name: "default.Table_032d8e14bf694d2fbcd3b37c7ef106f9" modify_external_catalogs: true
I20240503 14:12:18.266080 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
I20240503 14:12:18.272605 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet fc3f7c221cbc467dabfe5665ccd07d48
I20240503 14:12:18.273259 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3366802ae4a446d388f1679e5cc304fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.273986 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3366802ae4a446d388f1679e5cc304fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.275111 28331 tablet_service.cc:1467] Processing CreateTablet for tablet f67c4cb980b44e9397bea70363fe1acd (DEFAULT_TABLE table=default.Table_0453cf78e30f45ffba602e0ca759e01c [id=78467b90af7b4fc1a4bbe1e475b8f593]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.278867 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_717ce9be26654b1d9f243797798e0fa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.279580 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_717ce9be26654b1d9f243797798e0fa3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.280284 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.281615 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.282567 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:18.284063 28614 tablet_bootstrap.cc:492] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:18.284986 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.284723 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 0e680f189add4b5c96de4046321aad2d (table default.Table_2d7ecf8640f64d82908807b4c4d4fd7f [id=d1088a94cd0442c9bd9837bf9f0e1ff9]). Expected version 2 got 0
I20240503 14:12:18.286048 28322 tablet_service.cc:1467] Processing CreateTablet for tablet b8d8f09130c04698949adc45f8b071d7 (DEFAULT_TABLE table=default.Table_b9af7d02fc5a42a7a09ecbfb33775ccf [id=77641df744384e6ba7b6015a650a859e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.286302 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3f7c221cbc467dabfe5665ccd07d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.276880 28653 ts_tablet_manager.cc:1888] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:18.286955 28653 log.cc:1196] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0926e523ef454a2089f4d40a328c1c6d
I20240503 14:12:18.287745 28604 tablet_bootstrap.cc:492] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.288146 28604 ts_tablet_manager.cc:1367] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.156s user 0.043s sys 0.030s
W20240503 14:12:18.288839 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet cf810e84367f420b928a171790849445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf810e84367f420b928a171790849445 already in progress: creating tablet
I20240503 14:12:18.290612 28604 raft_consensus.cc:348] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.291393 28604 raft_consensus.cc:374] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.291764 28604 raft_consensus.cc:729] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.290791 28472 tablet_service.cc:1467] Processing CreateTablet for tablet f67c4cb980b44e9397bea70363fe1acd (DEFAULT_TABLE table=default.Table_0453cf78e30f45ffba602e0ca759e01c [id=78467b90af7b4fc1a4bbe1e475b8f593]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.292019 28614 ts_tablet_manager.cc:1367] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.287s user 0.000s sys 0.015s
I20240503 14:12:18.292842 28604 consensus_queue.cc:260] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.295713 28604 ts_tablet_manager.cc:1398] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:18.296589 28661 ts_tablet_manager.cc:1888] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.296938 28661 log.cc:1196] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/37dc7d96357743089372baf71654b1e3
I20240503 14:12:18.296993 28604 tablet_bootstrap.cc:492] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.297040 28614 raft_consensus.cc:348] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.297691 28614 raft_consensus.cc:374] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.297886 28614 raft_consensus.cc:729] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.298030 28185 tablet_service.cc:1467] Processing CreateTablet for tablet b8d8f09130c04698949adc45f8b071d7 (DEFAULT_TABLE table=default.Table_b9af7d02fc5a42a7a09ecbfb33775ccf [id=77641df744384e6ba7b6015a650a859e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.298503 28614 consensus_queue.cc:260] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.299374 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d8f09130c04698949adc45f8b071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.300155 28030 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:18.302242 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a5e1a1d0ff44431fa10153a63904f373"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.303143 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5e1a1d0ff44431fa10153a63904f373 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.305585 28604 tablet_bootstrap.cc:654] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.289737 28178 tablet_service.cc:1467] Processing CreateTablet for tablet f67c4cb980b44e9397bea70363fe1acd (DEFAULT_TABLE table=default.Table_0453cf78e30f45ffba602e0ca759e01c [id=78467b90af7b4fc1a4bbe1e475b8f593]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.307463 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67c4cb980b44e9397bea70363fe1acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.307618 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67c4cb980b44e9397bea70363fe1acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.311376 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6adeb488fd34301915f2977ce8cf61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.311642 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db657eef2db4e3791318e01e1a965fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.311899 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d8f09130c04698949adc45f8b071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.312110 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc004243d5a947e3a8852605b0414c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.312479 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bba058a8f147a1ae336b3be60a20b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.312914 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888c28447db64751adca55ae769d0a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.314719 29456 raft_consensus.cc:482] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.314880 29456 raft_consensus.cc:504] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.315692 29456 leader_election.cc:288] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:18.316223 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37364d5facc241f4bbccc979a50a0ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.316817 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d4e4a1f0b14406be957426a45bf45d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.317135 28489 raft_consensus.cc:2454] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:18.317786 28290 leader_election.cc:302] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.318204 29456 raft_consensus.cc:2781] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:18.318284 29456 raft_consensus.cc:482] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:18.318338 29456 raft_consensus.cc:3037] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.321429 29456 raft_consensus.cc:504] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.324172 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d4e4a1f0b14406be957426a45bf45d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:18.324416 28489 raft_consensus.cc:3037] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.327308 28489 raft_consensus.cc:2454] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:18.328138 28290 leader_election.cc:302] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.328480 29598 raft_consensus.cc:2781] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.328589 29598 raft_consensus.cc:686] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:18.328850 29598 consensus_queue.cc:237] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.332206 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67c4cb980b44e9397bea70363fe1acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.333515 29456 leader_election.cc:288] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:18.333976 28604 tablet_bootstrap.cc:492] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.334178 28604 ts_tablet_manager.cc:1367] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.000s
I20240503 14:12:18.333884 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc4677a01f14f7c87ca00b29c6238c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.334996 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a278bb9fe04a1a93c15da6cf38d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.335078 28604 raft_consensus.cc:348] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.335358 28604 raft_consensus.cc:374] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.335469 28604 raft_consensus.cc:729] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.335783 28604 consensus_queue.cc:260] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.336570 28604 ts_tablet_manager.cc:1398] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:18.336930 28604 tablet_bootstrap.cc:492] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:18.341385 28290 leader_election.cc:341] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b7d4e4a1f0b14406be957426a45bf45d
I20240503 14:12:18.342123 29456 consensus_queue.cc:1035] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:18.343096 28290 leader_election.cc:341] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b7d4e4a1f0b14406be957426a45bf45d
I20240503 14:12:18.343922 28604 tablet_bootstrap.cc:654] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:18.345764 29575 log.cc:924] Time spent T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Append to log took a long time: real 0.154s user 0.000s sys 0.000s
I20240503 14:12:18.351261 28661 ts_tablet_manager.cc:1909] T 37dc7d96357743089372baf71654b1e3 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:18.351639 28653 ts_tablet_manager.cc:1909] T 0926e523ef454a2089f4d40a328c1c6d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:18.352617 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 666) took 1290 ms. Trace:
I20240503 14:12:18.352993 28653 rpcz_store.cc:276] 0503 14:12:17.062270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.114315 (+ 52045us) service_pool.cc:225] Handling call
0503 14:12:18.044383 (+930068us) ts_tablet_manager.cc:1120] Deleting tablet 0926e523ef454a2089f4d40a328c1c6d
0503 14:12:18.044399 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.092541 (+ 48142us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.275782 (+183241us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.352535 (+ 76753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":904501}
I20240503 14:12:18.354564 28653 tablet_replica.cc:330] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:18.355118 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 639) took 2121 ms. Trace:
I20240503 14:12:18.355160 28653 raft_consensus.cc:2229] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.355365 28661 rpcz_store.cc:276] 0503 14:12:16.233550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.233680 (+ 130us) service_pool.cc:225] Handling call
0503 14:12:18.097996 (+1864316us) ts_tablet_manager.cc:1120] Deleting tablet 37dc7d96357743089372baf71654b1e3
0503 14:12:18.098007 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.144347 (+ 46340us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.296352 (+152005us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.355084 (+ 58732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792805}
I20240503 14:12:18.358618 28661 tablet_replica.cc:330] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:18.358606 28653 raft_consensus.cc:2258] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.359431 28661 raft_consensus.cc:2229] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.360442 28661 raft_consensus.cc:2258] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.361665 28600 tablet_bootstrap.cc:654] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.366287 28661 ts_tablet_manager.cc:1875] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.380620 29701 raft_consensus.cc:482] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.380826 29701 raft_consensus.cc:504] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.381567 29701 leader_election.cc:288] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.382692 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b52ea336524beea2f255c8720f7472" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.382946 28353 raft_consensus.cc:2454] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:18.383075 28427 leader_election.cc:341] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 00b52ea336524beea2f255c8720f7472
I20240503 14:12:18.383618 28427 leader_election.cc:302] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:18.384106 29602 raft_consensus.cc:482] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.384189 29701 raft_consensus.cc:2781] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:18.384265 29701 raft_consensus.cc:482] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:18.384326 29701 raft_consensus.cc:3037] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.384305 29602 raft_consensus.cc:504] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.385015 29602 leader_election.cc:288] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:18.386972 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 903f11a608b1483887d9b5234a404470 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e7b2865e9604ce58280321c7ee2a88e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.386974 28462 tablet_service.cc:1467] Processing CreateTablet for tablet b8d8f09130c04698949adc45f8b071d7 (DEFAULT_TABLE table=default.Table_b9af7d02fc5a42a7a09ecbfb33775ccf [id=77641df744384e6ba7b6015a650a859e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.387317 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.387471 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d8f09130c04698949adc45f8b071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.387791 28464 tablet_service.cc:1467] Processing CreateTablet for tablet dabb44f07d0343cbb4f87282e19c7f7f (DEFAULT_TABLE table=default.Table_871098e8af8c4007bb6ec13983c587de [id=0e8af363df974e6eb6fe9203fb46b8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.388228 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabb44f07d0343cbb4f87282e19c7f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.388466 28187 tablet_service.cc:1467] Processing CreateTablet for tablet dabb44f07d0343cbb4f87282e19c7f7f (DEFAULT_TABLE table=default.Table_871098e8af8c4007bb6ec13983c587de [id=0e8af363df974e6eb6fe9203fb46b8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.388912 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabb44f07d0343cbb4f87282e19c7f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.388764 28653 ts_tablet_manager.cc:1875] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.392138 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 816812043a2842f4aa3278835f37e553 (DEFAULT_TABLE table=default.Table_986bd23778564daa8b25045aa3d81420 [id=7656dfa5242742e28b10c04fac6981a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.392653 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816812043a2842f4aa3278835f37e553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.393033 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 816812043a2842f4aa3278835f37e553 (DEFAULT_TABLE table=default.Table_986bd23778564daa8b25045aa3d81420 [id=7656dfa5242742e28b10c04fac6981a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.393501 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816812043a2842f4aa3278835f37e553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.396610 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903f11a608b1483887d9b5234a404470" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:18.397301 28150 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.397743 29701 raft_consensus.cc:504] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.398761 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903f11a608b1483887d9b5234a404470" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.399755 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3f7c221cbc467dabfe5665ccd07d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.400034 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b52ea336524beea2f255c8720f7472" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:18.400275 28353 raft_consensus.cc:3037] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:18.400342 28150 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.400498 28150 leader_election.cc:302] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:18.401304 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 816812043a2842f4aa3278835f37e553 (DEFAULT_TABLE table=default.Table_986bd23778564daa8b25045aa3d81420 [id=7656dfa5242742e28b10c04fac6981a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.401381 29602 raft_consensus.cc:2726] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.401899 28327 tablet_service.cc:1467] Processing CreateTablet for tablet dabb44f07d0343cbb4f87282e19c7f7f (DEFAULT_TABLE table=default.Table_871098e8af8c4007bb6ec13983c587de [id=0e8af363df974e6eb6fe9203fb46b8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.401883 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 37dc7d96357743089372baf71654b1e3 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]) successfully deleted
I20240503 14:12:18.402444 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 4451a8c3ab1e4fccac7100ac74bb990f (DEFAULT_TABLE table=default.Table_a6b8bd2eddaf402f992d809054a1bae3 [id=f632a2b3ec204ee5b2baad153068b76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.402871 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 070322ca6988449e97c2f25a0dbc671f (DEFAULT_TABLE table=default.Table_fb411e327c414a45b69f242c2113111f [id=bf2496910d274254b89d09228a82c60a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.403944 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 4451a8c3ab1e4fccac7100ac74bb990f (DEFAULT_TABLE table=default.Table_a6b8bd2eddaf402f992d809054a1bae3 [id=f632a2b3ec204ee5b2baad153068b76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.404054 29700 raft_consensus.cc:482] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.404508 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4451a8c3ab1e4fccac7100ac74bb990f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.404451 29700 raft_consensus.cc:504] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.405169 29700 leader_election.cc:288] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.407310 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:18.407660 28212 raft_consensus.cc:2379] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:18.408151 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:18.408361 28427 consensus_peers.cc:485] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1603e2360041427f8635782e812b6431. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:18.408941 28427 leader_election.cc:302] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.409459 29700 raft_consensus.cc:3037] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.411928 29700 raft_consensus.cc:2726] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.306915 28614 ts_tablet_manager.cc:1398] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:12:18.413445 28614 tablet_bootstrap.cc:492] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:18.413053 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet bf21f6234dd64cf8a32f1599fe50746e with delete_type TABLET_DATA_DELETED (replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.413641 28353 raft_consensus.cc:2454] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:18.416767 28427 leader_election.cc:302] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.417287 29700 raft_consensus.cc:2781] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.417467 29700 raft_consensus.cc:686] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:18.417748 29700 consensus_queue.cc:237] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.419122 29701 leader_election.cc:288] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:18.420531 28427 leader_election.cc:341] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 00b52ea336524beea2f255c8720f7472
I20240503 14:12:18.422703 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3f7c221cbc467dabfe5665ccd07d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.426342 28489 raft_consensus.cc:3037] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.430620 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]) successfully deleted
I20240503 14:12:18.432207 28264 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:18.432925 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070322ca6988449e97c2f25a0dbc671f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.433784 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816812043a2842f4aa3278835f37e553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.434402 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4451a8c3ab1e4fccac7100ac74bb990f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.440243 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabb44f07d0343cbb4f87282e19c7f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.445829 28604 tablet_bootstrap.cc:492] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.446383 28604 ts_tablet_manager.cc:1367] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.109s user 0.021s sys 0.008s
I20240503 14:12:18.448204 28604 raft_consensus.cc:348] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.448674 28604 raft_consensus.cc:374] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.448843 28604 raft_consensus.cc:729] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.449191 28604 consensus_queue.cc:260] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.450052 28604 ts_tablet_manager.cc:1398] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:18.450532 28604 tablet_bootstrap.cc:492] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.452522 28604 tablet_bootstrap.cc:654] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:18.456434 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet fc3f7c221cbc467dabfe5665ccd07d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc3f7c221cbc467dabfe5665ccd07d48 already in progress: creating tablet
I20240503 14:12:18.460512 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 070322ca6988449e97c2f25a0dbc671f (DEFAULT_TABLE table=default.Table_fb411e327c414a45b69f242c2113111f [id=bf2496910d274254b89d09228a82c60a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.466331 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 4451a8c3ab1e4fccac7100ac74bb990f (DEFAULT_TABLE table=default.Table_a6b8bd2eddaf402f992d809054a1bae3 [id=f632a2b3ec204ee5b2baad153068b76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.467717 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4451a8c3ab1e4fccac7100ac74bb990f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.467876 28614 tablet_bootstrap.cc:654] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.468191 29602 raft_consensus.cc:482] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.473242 29595 consensus_queue.cc:1035] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.481037 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070322ca6988449e97c2f25a0dbc671f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.468494 29602 raft_consensus.cc:504] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.485114 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c5dfac02d4dfe932996d2a37fe031" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.485769 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 070322ca6988449e97c2f25a0dbc671f (DEFAULT_TABLE table=default.Table_fb411e327c414a45b69f242c2113111f [id=bf2496910d274254b89d09228a82c60a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.487877 29602 leader_election.cc:288] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:18.488149 29602 raft_consensus.cc:482] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.488368 29602 raft_consensus.cc:504] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.489265 29602 leader_election.cc:288] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:18.490213 28150 leader_election.cc:341] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.490497 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af0342e0e0b47499a9322b5b2cb3d03" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.490466 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af0342e0e0b47499a9322b5b2cb3d03" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:18.491096 28150 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:18.491384 28150 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.491367 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c5dfac02d4dfe932996d2a37fe031" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.491570 28150 leader_election.cc:302] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:18.492151 28150 leader_election.cc:341] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.492316 28150 leader_election.cc:302] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:18.492810 29602 raft_consensus.cc:2726] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.493048 29602 raft_consensus.cc:2726] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.495299 29700 raft_consensus.cc:482] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.498819 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 332ecc478e49424d9e7d7decbbc2c29c (DEFAULT_TABLE table=default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea [id=23231f74e45e4d3484db472d381f97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.499446 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332ecc478e49424d9e7d7decbbc2c29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.505779 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070322ca6988449e97c2f25a0dbc671f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.505656 28600 tablet_bootstrap.cc:492] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:18.511569 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 332ecc478e49424d9e7d7decbbc2c29c (DEFAULT_TABLE table=default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea [id=23231f74e45e4d3484db472d381f97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.515867 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332ecc478e49424d9e7d7decbbc2c29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.519842 29700 raft_consensus.cc:504] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.523680 28026 catalog_manager.cc:5484] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:18.524379 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:18.524770 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:18.525185 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a3023785b11c4ad7b8757a715f0952c5 (table default.Table_facfcbde0b2740359615d4907998c7fa [id=750bb7e271c241d2bbf14bd91dbc42d0]). Expected version 1 got 0
I20240503 14:12:18.525424 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:18.525614 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
I20240503 14:12:18.525861 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet df63f75c157a4e928d4dbbda06cfa5ce (table default.Table_13b6a3c3a46b4308b4ce309165333325 [id=d29fe68058a14e178e42a10043a91738]). Expected version 2 got 0
I20240503 14:12:18.530624 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.531128 28603 tablet_replica.cc:330] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:18.531347 28603 raft_consensus.cc:2229] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.531522 28603 raft_consensus.cc:2258] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.532212 28603 ts_tablet_manager.cc:1875] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:18.533807 28427 leader_election.cc:341] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 06fb79fcf30d485798c64c815e372dc1
I20240503 14:12:18.533452 29700 leader_election.cc:288] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:18.539602 28603 ts_tablet_manager.cc:1888] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.539876 28603 log.cc:1196] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/b25c852aa7aa4fbfafdacec43f5bc426
I20240503 14:12:18.540464 28603 ts_tablet_manager.cc:1909] T b25c852aa7aa4fbfafdacec43f5bc426 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:18.541237 29730 raft_consensus.cc:482] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.541568 29730 raft_consensus.cc:504] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:18.542645 28427 leader_election.cc:341] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 06fb79fcf30d485798c64c815e372dc1
I20240503 14:12:18.542797 28427 leader_election.cc:302] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.542871 29730 leader_election.cc:288] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:18.520908 28661 ts_tablet_manager.cc:1888] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.543098 28661 log.cc:1196] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/1d77014c66624f958b38f7cc83644891
I20240503 14:12:18.543473 28661 ts_tablet_manager.cc:1909] T 1d77014c66624f958b38f7cc83644891 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:18.544497 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 640) took 2310 ms. Trace:
I20240503 14:12:18.544781 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f0081cc2cb48aaa368741109032fd1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.545109 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1728b6f0e64661aaa8184ce51366d2 with delete_type TABLET_DATA_DELETED (replaced by tablet d4c8a97076a44f9eaa3a15026f90a304) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.546125 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet b25c852aa7aa4fbfafdacec43f5bc426 (table default.Table_3e4ec426791c4b11b43d3c36260eedd4 [id=c14f7d47ddd64652aa553945f775aec6]) successfully deleted
W20240503 14:12:18.546453 28150 leader_election.cc:341] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.549790 28614 tablet_bootstrap.cc:492] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:18.549988 28614 ts_tablet_manager.cc:1367] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.137s user 0.005s sys 0.001s
I20240503 14:12:18.550992 28614 raft_consensus.cc:348] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.551227 28614 raft_consensus.cc:374] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.551290 28614 raft_consensus.cc:729] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.551105 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 332ecc478e49424d9e7d7decbbc2c29c (DEFAULT_TABLE table=default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea [id=23231f74e45e4d3484db472d381f97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.552634 28614 consensus_queue.cc:260] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.554378 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:18.555099 28614 ts_tablet_manager.cc:1398] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:18.555444 28614 tablet_bootstrap.cc:492] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:18.555739 28600 ts_tablet_manager.cc:1367] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.396s user 0.006s sys 0.007s
W20240503 14:12:18.557173 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 8b1728b6f0e64661aaa8184ce51366d2 because the tablet was not found. No further retry: Not found: Tablet not found: 8b1728b6f0e64661aaa8184ce51366d2
I20240503 14:12:18.544714 29701 raft_consensus.cc:2726] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.544929 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f0081cc2cb48aaa368741109032fd1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.557705 28661 rpcz_store.cc:276] 0503 14:12:16.233723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.233889 (+ 166us) service_pool.cc:225] Handling call
0503 14:12:18.358545 (+2124656us) ts_tablet_manager.cc:1120] Deleting tablet 1d77014c66624f958b38f7cc83644891
0503 14:12:18.358561 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.479302 (+120741us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.520770 (+ 41468us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.544470 (+ 23700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2053295}
I20240503 14:12:18.545120 29700 raft_consensus.cc:482] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.558744 29700 raft_consensus.cc:504] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.559476 29700 leader_election.cc:288] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:18.552304 28427 consensus_peers.cc:485] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1603e2360041427f8635782e812b6431. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:18.560387 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.560817 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 1d77014c66624f958b38f7cc83644891 (table default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]) successfully deleted
I20240503 14:12:18.558218 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 636) took 2325 ms. Trace:
I20240503 14:12:18.562471 28661 rpcz_store.cc:276] 0503 14:12:16.232261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.237080 (+ 4819us) service_pool.cc:225] Handling call
0503 14:12:18.558039 (+2320959us) ts_tablet_manager.cc:1120] Deleting tablet a1ebec4622ef4950bd1c70451958c59b
0503 14:12:18.558053 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.558194 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2252749}
W20240503 14:12:18.559733 28150 leader_election.cc:341] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.560467 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
W20240503 14:12:18.562897 28427 leader_election.cc:341] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.565667 28150 leader_election.cc:302] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:18.566488 29730 raft_consensus.cc:2726] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:18.567116 28427 leader_election.cc:341] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.567270 28427 leader_election.cc:302] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.567312 28661 tablet_replica.cc:330] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:18.567530 28661 raft_consensus.cc:2229] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.567713 28661 raft_consensus.cc:2258] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.567914 29701 raft_consensus.cc:2726] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.573107 28491 raft_consensus.cc:1261] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Refusing update from remote peer d3a6b9b7913444d29c96581af6febf33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:18.573450 28214 raft_consensus.cc:1261] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Refusing update from remote peer d3a6b9b7913444d29c96581af6febf33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:18.573969 29456 consensus_queue.cc:1035] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.574553 29456 consensus_queue.cc:1035] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.579267 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b52ea336524beea2f255c8720f7472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.582559 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet b25c852aa7aa4fbfafdacec43f5bc426 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.583432 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 5775006e9f8f4612aa70dfb316de9ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:05 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.584208 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 140de4d395d54b7696062ed96b8aa89c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.584160 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332ecc478e49424d9e7d7decbbc2c29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.590428 29595 consensus_queue.cc:1035] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.598745 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6b74a6afc6cf44d19931b689292e8a36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.603047 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b74a6afc6cf44d19931b689292e8a36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:18.601812 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c8bf34e96e04868b135a1a9422a46be consensus state has no leader
W20240503 14:12:18.603685 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f91ddbc250c44fbfbf4ffe83433d8dd8 has no consensus state
I20240503 14:12:18.604378 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_bceb92591b5d4af8bf691e736575984d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:18.605223 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b2fcbb86236149c6865ae80f54e4cac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:18.605938 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet a6b468dd4102495abd4337edcfc6340a from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:18.606643 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2fcbb86236149c6865ae80f54e4cac3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.603283 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2d5192083ed94a3d8d2d1a0bf001a1eb
I20240503 14:12:18.611034 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0453cf78e30f45ffba602e0ca759e01c" } modify_external_catalogs: true
I20240503 14:12:18.611275 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.611316 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0453cf78e30f45ffba602e0ca759e01c" } modify_external_catalogs: true
W20240503 14:12:18.612067 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d5192083ed94a3d8d2d1a0bf001a1eb already in progress: creating tablet
I20240503 14:12:18.612529 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:18.605423 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bceb92591b5d4af8bf691e736575984d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.614607 28614 tablet_bootstrap.cc:654] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.608024 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_461c1f89931e46ed9efa3024088d2adb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.616230 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_461c1f89931e46ed9efa3024088d2adb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.600872 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a0610cdc38c7464888c7b7366ed72c2f" } new_table_name: "default.Table_2caebbf3e5894625ada675783f21a881" modify_external_catalogs: true
I20240503 14:12:18.602299 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:18.623294 28027 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet e6c90869b5034128864faaf05697c230 (table default.Table_5dcdff74f1f7407084a45dc9fe588103 [id=47ade8624b00424d8fc04dbca5174ff4]). Expected version 3 got 0
I20240503 14:12:18.623400 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3bf21cdd53304e70a9c5b910bc33ce19" } modify_external_catalogs: true
I20240503 14:12:18.623795 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3bf21cdd53304e70a9c5b910bc33ce19" } modify_external_catalogs: true
I20240503 14:12:18.623862 28027 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:18.624369 28027 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:18.628630 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.632158 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dcf7f653c09f4112b516cee572ea9993" } modify_external_catalogs: true
I20240503 14:12:18.633661 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:18.634243 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a6b468dd4102495abd4337edcfc6340a
I20240503 14:12:18.635047 28031 catalog_manager.cc:6554] ReplaceTablet: tablet a6b468dd4102495abd4337edcfc6340a deleted and replaced by tablet f783ee62adba49579a0518b240b795bc
I20240503 14:12:18.635926 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b468dd4102495abd4337edcfc6340a with delete_type TABLET_DATA_DELETED (replaced by tablet f783ee62adba49579a0518b240b795bc) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.638044 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b468dd4102495abd4337edcfc6340a with delete_type TABLET_DATA_DELETED (replaced by tablet f783ee62adba49579a0518b240b795bc) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.636822 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5a4ea581b23745598fabca5d5e715f73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.639250 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a6b468dd4102495abd4337edcfc6340a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b468dd4102495abd4337edcfc6340a already in progress: creating tablet
W20240503 14:12:18.639991 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a4ea581b23745598fabca5d5e715f73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.640223 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f67c4cb980b44e9397bea70363fe1acd
I20240503 14:12:18.642138 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet f67c4cb980b44e9397bea70363fe1acd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.642549 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet f67c4cb980b44e9397bea70363fe1acd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:18.643757 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f67c4cb980b44e9397bea70363fe1acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f67c4cb980b44e9397bea70363fe1acd already in progress: creating tablet
I20240503 14:12:18.644063 28026 catalog_manager.cc:5484] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:18.644770 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9f158ace156f44beacdd67afbd7eafc0" } new_table_name: "default.Table_9af4e42a49be4a0a877834654f2171af" modify_external_catalogs: true
I20240503 14:12:18.646615 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b509f93207e3417c9bfe50c97d92096e" } new_table_name: "default.Table_b07a739f381c48e28998d3e15695f45f" modify_external_catalogs: true
I20240503 14:12:18.648674 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7dc512798f934fd4a2d847c3f9ae1c16" } modify_external_catalogs: true
I20240503 14:12:18.648942 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7dc512798f934fd4a2d847c3f9ae1c16" } modify_external_catalogs: true
I20240503 14:12:18.650328 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_dcf7f653c09f4112b516cee572ea9993" } modify_external_catalogs: true
W20240503 14:12:18.651005 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6eab5a52d6354bcdbd507741f06f21cf consensus state has no leader
I20240503 14:12:18.652360 28661 ts_tablet_manager.cc:1875] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.657332 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
I20240503 14:12:18.658071 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:18.665428 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0a6a58db50f849a8be729c10dbe03e76
I20240503 14:12:18.665690 28604 tablet_bootstrap.cc:492] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.665917 28604 ts_tablet_manager.cc:1367] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.216s user 0.038s sys 0.033s
I20240503 14:12:18.667052 28604 raft_consensus.cc:348] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.667685 28604 raft_consensus.cc:374] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.668001 28604 raft_consensus.cc:729] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.668612 28604 consensus_queue.cc:260] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.669364 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a58db50f849a8be729c10dbe03e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.669142 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a58db50f849a8be729c10dbe03e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:18.670544 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0a6a58db50f849a8be729c10dbe03e76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6a58db50f849a8be729c10dbe03e76 already in progress: creating tablet
I20240503 14:12:18.670920 29736 tablet.cc:1722] T a3023785b11c4ad7b8757a715f0952c5 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:18.672677 29735 tablet.cc:1722] T a3023785b11c4ad7b8757a715f0952c5 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:18.679616 28614 tablet_bootstrap.cc:492] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:18.679898 28614 ts_tablet_manager.cc:1367] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20240503 14:12:18.680747 29730 raft_consensus.cc:482] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.681246 29730 raft_consensus.cc:504] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.682160 29730 leader_election.cc:288] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:18.680912 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet f67c4cb980b44e9397bea70363fe1acd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.680931 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.680948 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b468dd4102495abd4337edcfc6340a with delete_type TABLET_DATA_DELETED (replaced by tablet f783ee62adba49579a0518b240b795bc) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.683331 29700 raft_consensus.cc:482] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.683528 29700 raft_consensus.cc:504] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.684171 28614 raft_consensus.cc:348] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.684326 29700 leader_election.cc:288] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.684387 28614 raft_consensus.cc:374] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.684448 28614 raft_consensus.cc:729] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.684731 28614 consensus_queue.cc:260] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.685475 28614 ts_tablet_manager.cc:1398] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:18.686483 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a58db50f849a8be729c10dbe03e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.686880 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a3023785b11c4ad7b8757a715f0952c5 (table default.Table_facfcbde0b2740359615d4907998c7fa [id=750bb7e271c241d2bbf14bd91dbc42d0]). Expected version 1 got 0
I20240503 14:12:18.687060 28600 raft_consensus.cc:348] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.687992 28600 raft_consensus.cc:374] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.688172 28600 raft_consensus.cc:729] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.688421 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df63f75c157a4e928d4dbbda06cfa5ce" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:18.688498 28600 consensus_queue.cc:260] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.688730 28217 raft_consensus.cc:2454] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:18.689567 28600 ts_tablet_manager.cc:1398] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.116s user 0.001s sys 0.001s
I20240503 14:12:18.690078 28600 tablet_bootstrap.cc:492] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:18.690639 28427 leader_election.cc:302] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.691138 29700 raft_consensus.cc:2781] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:18.691222 29700 raft_consensus.cc:482] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:18.691215 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dd4c914a4846bab3a00408247ec64a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.691365 29700 raft_consensus.cc:3037] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.686422 28614 tablet_bootstrap.cc:492] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:18.694075 29700 raft_consensus.cc:504] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.687430 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df63f75c157a4e928d4dbbda06cfa5ce" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.695235 28353 raft_consensus.cc:2454] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:18.695300 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df63f75c157a4e928d4dbbda06cfa5ce" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:18.695482 28351 raft_consensus.cc:3037] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.687516 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dd4c914a4846bab3a00408247ec64a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.696902 29700 leader_election.cc:288] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.703336 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet e69e01bdbae349d0a61a295b6f6352f0 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:18.704491 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:18.707101 28661 ts_tablet_manager.cc:1888] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.708061 28661 log.cc:1196] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0982b99a02ee48f2b6379e7be56c4c02
I20240503 14:12:18.710614 28661 ts_tablet_manager.cc:1909] T 0982b99a02ee48f2b6379e7be56c4c02 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:18.715875 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 641) took 2479 ms. Trace:
W20240503 14:12:18.720245 28150 leader_election.cc:341] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.720245 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df63f75c157a4e928d4dbbda06cfa5ce" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
W20240503 14:12:18.720746 28150 leader_election.cc:341] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.720885 28217 raft_consensus.cc:3037] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.721540 29700 consensus_queue.cc:1035] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.723330 28604 ts_tablet_manager.cc:1398] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
I20240503 14:12:18.723713 28604 tablet_bootstrap.cc:492] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:18.731431 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 137c9eef465843049e93374367e562af consensus state has no leader
I20240503 14:12:18.732280 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7927f3e0fb214441aaf52352b6956ef6
I20240503 14:12:18.732839 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ac577af6460b423288a790e532b00f82
I20240503 14:12:18.734306 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet ac577af6460b423288a790e532b00f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.734306 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 7927f3e0fb214441aaf52352b6956ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.734781 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet ac577af6460b423288a790e532b00f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.735376 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 7927f3e0fb214441aaf52352b6956ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:18.735718 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ac577af6460b423288a790e532b00f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac577af6460b423288a790e532b00f82 already in progress: creating tablet
W20240503 14:12:18.736204 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7927f3e0fb214441aaf52352b6956ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7927f3e0fb214441aaf52352b6956ef6 already in progress: creating tablet
W20240503 14:12:18.736760 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bc05dfe2b324ca787ca33f52be6a308 consensus state has no leader
I20240503 14:12:18.738139 28614 tablet_bootstrap.cc:654] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.716871 28661 rpcz_store.cc:276] 0503 14:12:16.236774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.237072 (+ 298us) service_pool.cc:225] Handling call
0503 14:12:18.567251 (+2330179us) ts_tablet_manager.cc:1120] Deleting tablet 0982b99a02ee48f2b6379e7be56c4c02
0503 14:12:18.567264 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.674508 (+107244us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.706144 (+ 31636us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.715718 (+ 9574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2261917}
I20240503 14:12:18.744576 28604 tablet_bootstrap.cc:654] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.744545 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0982b99a02ee48f2b6379e7be56c4c02 (table default.Table_79726aaee899484aaf8147ce414cbd87 [id=8cce76d1e8a642a4b051b4c4a854be78]) successfully deleted
I20240503 14:12:18.749547 28661 tablet_replica.cc:330] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:18.749888 28661 raft_consensus.cc:2229] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.750934 29595 consensus_queue.cc:1035] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.750036 29700 consensus_queue.cc:1035] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.751914 28661 raft_consensus.cc:2258] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.754298 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b52ea336524beea2f255c8720f7472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.754599 28603 tablet_replica.cc:330] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:18.754782 28603 raft_consensus.cc:2229] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.754932 28603 raft_consensus.cc:2258] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.755378 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d4e4a1f0b14406be957426a45bf45d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.755663 28603 ts_tablet_manager.cc:1875] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.756596 28661 ts_tablet_manager.cc:1875] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.759732 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e69e01bdbae349d0a61a295b6f6352f0
I20240503 14:12:18.760571 28029 catalog_manager.cc:6554] ReplaceTablet: tablet e69e01bdbae349d0a61a295b6f6352f0 deleted and replaced by tablet 29c78b82bd264b53ba60583e6d8f06d4
I20240503 14:12:18.764504 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet e69e01bdbae349d0a61a295b6f6352f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 29c78b82bd264b53ba60583e6d8f06d4) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:18.764734 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet e69e01bdbae349d0a61a295b6f6352f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 29c78b82bd264b53ba60583e6d8f06d4) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:18.765794 28614 tablet_bootstrap.cc:492] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:18.766188 28614 ts_tablet_manager.cc:1367] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.000s
I20240503 14:12:18.768016 28614 raft_consensus.cc:348] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.770179 28614 raft_consensus.cc:374] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.770601 28614 raft_consensus.cc:729] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.771296 28614 consensus_queue.cc:260] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.774776 28614 ts_tablet_manager.cc:1398] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:18.776321 28614 tablet_bootstrap.cc:492] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:18.772881 28603 ts_tablet_manager.cc:1888] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.778391 28603 log.cc:1196] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/00b52ea336524beea2f255c8720f7472
I20240503 14:12:18.779110 28603 ts_tablet_manager.cc:1909] T 00b52ea336524beea2f255c8720f7472 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:18.720888 28150 leader_election.cc:302] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:18.783747 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 6809b59e87644316b9a450d6b670da2b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:18.784229 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 00b52ea336524beea2f255c8720f7472 (table default.Table_9bcbc7e8d1154ffd94951feb50445bc2 [id=eb3d80381dc844dcac4ed2bc171565d0]) successfully deleted
I20240503 14:12:18.784510 28603 tablet_replica.cc:330] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:18.784551 28653 ts_tablet_manager.cc:1888] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:18.785764 28603 raft_consensus.cc:2229] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:18.786403 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet ac577af6460b423288a790e532b00f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.787235 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 7927f3e0fb214441aaf52352b6956ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.787580 28604 tablet_bootstrap.cc:492] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.787740 28603 pending_rounds.cc:62] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:18.787829 28604 ts_tablet_manager.cc:1367] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.000s
I20240503 14:12:18.788177 28603 pending_rounds.cc:68] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597724997816320 op_type: NO_OP noop_request { }
I20240503 14:12:18.788758 28603 raft_consensus.cc:2866] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:18.788879 28604 raft_consensus.cc:348] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.789242 28604 raft_consensus.cc:374] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.789345 28604 raft_consensus.cc:729] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.789675 28604 consensus_queue.cc:260] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.794441 29602 raft_consensus.cc:2726] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.794982 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet e69e01bdbae349d0a61a295b6f6352f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 29c78b82bd264b53ba60583e6d8f06d4) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:18.797050 28653 log.cc:1196] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a32b6995818041778617a0d6bf936c8c
I20240503 14:12:18.797364 28217 raft_consensus.cc:2454] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:18.797585 28653 ts_tablet_manager.cc:1909] T a32b6995818041778617a0d6bf936c8c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:18.797684 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 63170.4us
I20240503 14:12:18.790524 28604 ts_tablet_manager.cc:1398] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:18.798607 28427 leader_election.cc:302] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:18.798972 29700 raft_consensus.cc:2781] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.799099 29700 raft_consensus.cc:686] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:18.799330 29700 consensus_queue.cc:237] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.801546 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9dbad41900f744188c2984de6c2db121"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.802306 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dbad41900f744188c2984de6c2db121 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.802232 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6a788c3137364f1bbfef7c497254ecbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.802803 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a788c3137364f1bbfef7c497254ecbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.803354 28603 raft_consensus.cc:2258] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.787001 28614 tablet_bootstrap.cc:654] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.804684 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0a52cc1bc610449fba51167cb16d9be8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:18.805034 28603 ts_tablet_manager.cc:1875] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:18.805253 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a52cc1bc610449fba51167cb16d9be8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.807992 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_13b6a3c3a46b4308b4ce309165333325" } new_table_name: "default.Table_e9e796aab9294a76b536ba18bd97568c" modify_external_catalogs: true
I20240503 14:12:18.809208 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f624029ec1b34039975e1aab597f5a1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:18.809588 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9bcc2ed2ad6745709f5987491f698eee" } new_table_name: "default.Table_1d16943aa7e24a50a783b77a82fdcc62" modify_external_catalogs: true
W20240503 14:12:18.809835 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f624029ec1b34039975e1aab597f5a1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.811213 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 670) took 1617 ms. Trace:
I20240503 14:12:18.811375 28653 rpcz_store.cc:276] 0503 14:12:17.194139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.194290 (+ 151us) service_pool.cc:225] Handling call
0503 14:12:18.354346 (+1160056us) ts_tablet_manager.cc:1120] Deleting tablet a32b6995818041778617a0d6bf936c8c
0503 14:12:18.354363 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.414201 (+ 59838us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.564252 (+150051us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.811130 (+246878us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159756}
I20240503 14:12:18.812359 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]) successfully deleted
I20240503 14:12:18.812901 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4fba6a05d4a44929a210ee59b0d60ccf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.813539 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fba6a05d4a44929a210ee59b0d60ccf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.820734 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_79df8c9603ae456799a73fb5badd18e9" } modify_external_catalogs: true
I20240503 14:12:18.821080 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_79df8c9603ae456799a73fb5badd18e9" } modify_external_catalogs: true
I20240503 14:12:18.824213 29701 raft_consensus.cc:482] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.824353 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 679) took 1382 ms. Trace:
I20240503 14:12:18.824391 29701 raft_consensus.cc:504] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.824523 28653 rpcz_store.cc:276] 0503 14:12:17.442177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.442273 (+ 96us) service_pool.cc:225] Handling call
0503 14:12:18.811719 (+1369446us) ts_tablet_manager.cc:1120] Deleting tablet 16fed718124a4640967ab13a1ac2b379
0503 14:12:18.824065 (+ 12346us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.824265 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369192}
I20240503 14:12:18.824756 28600 tablet_bootstrap.cc:654] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.825049 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 680) took 1382 ms. Trace:
I20240503 14:12:18.825099 29701 leader_election.cc:288] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.825178 28653 rpcz_store.cc:276] 0503 14:12:17.443015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.443076 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:18.824817 (+1381741us) ts_tablet_manager.cc:1120] Deleting tablet 11d9f47543524fc5921c99aad869035f
0503 14:12:18.824827 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.824968 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381528}
I20240503 14:12:18.825939 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.826467 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 681) took 1381 ms. Trace:
W20240503 14:12:18.826918 28427 leader_election.cc:341] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:18.826915 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 16fed718124a4640967ab13a1ac2b379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16fed718124a4640967ab13a1ac2b379 already in progress: creating tablet
I20240503 14:12:18.827296 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
W20240503 14:12:18.827451 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 11d9f47543524fc5921c99aad869035f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11d9f47543524fc5921c99aad869035f already in progress: creating tablet
I20240503 14:12:18.827643 28653 rpcz_store.cc:276] 0503 14:12:17.443670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.443716 (+ 46us) service_pool.cc:225] Handling call
0503 14:12:18.825405 (+1381689us) ts_tablet_manager.cc:1120] Deleting tablet e21a5dafa82f4660927d96c779a23478
0503 14:12:18.825416 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.825510 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381524}
W20240503 14:12:18.827934 28427 leader_election.cc:341] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.828050 28427 leader_election.cc:302] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.828315 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 678) took 1386 ms. Trace:
I20240503 14:12:18.828331 29701 raft_consensus.cc:2726] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.828439 28653 rpcz_store.cc:276] 0503 14:12:17.441729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.445470 (+ 3741us) service_pool.cc:225] Handling call
0503 14:12:18.828065 (+1382595us) ts_tablet_manager.cc:1120] Deleting tablet 007c84fd9d9247728b0babd48689aba8
0503 14:12:18.828077 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.828257 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1382113}
I20240503 14:12:18.828814 28653 tablet_replica.cc:330] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:18.829087 28653 raft_consensus.cc:2229] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:18.830206 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e21a5dafa82f4660927d96c779a23478 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e21a5dafa82f4660927d96c779a23478 already in progress: creating tablet
W20240503 14:12:18.830601 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 007c84fd9d9247728b0babd48689aba8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 007c84fd9d9247728b0babd48689aba8 already in progress: creating tablet
I20240503 14:12:18.835547 28653 raft_consensus.cc:2258] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.836017 29602 raft_consensus.cc:482] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.836324 28604 tablet_bootstrap.cc:492] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:18.836270 29602 raft_consensus.cc:504] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.837152 29602 leader_election.cc:288] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:18.838150 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf21f6234dd64cf8a32f1599fe50746e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:18.838996 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf21f6234dd64cf8a32f1599fe50746e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:18.839848 28653 ts_tablet_manager.cc:1875] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:18.840013 28150 leader_election.cc:341] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:18.840553 28150 leader_election.cc:341] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.840715 28150 leader_election.cc:302] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:18.841420 29602 raft_consensus.cc:2726] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.843127 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_afadb9c7ad4a4debb06a243990cbc46e" } modify_external_catalogs: true
I20240503 14:12:18.843395 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_afadb9c7ad4a4debb06a243990cbc46e" } modify_external_catalogs: true
I20240503 14:12:18.844521 29701 raft_consensus.cc:482] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.844692 29701 raft_consensus.cc:504] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.845420 29701 leader_election.cc:288] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:18.845505 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_02def817ba4f4e739e340bed3ea01b0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.846236 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02def817ba4f4e739e340bed3ea01b0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.846429 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140de4d395d54b7696062ed96b8aa89c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:18.847084 28427 leader_election.cc:341] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:18.849920 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140de4d395d54b7696062ed96b8aa89c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:18.852784 28661 ts_tablet_manager.cc:1888] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.852886 29746 tablet.cc:1722] T a3023785b11c4ad7b8757a715f0952c5 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 14:12:18.854544 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df63f75c157a4e928d4dbbda06cfa5ce consensus state has no leader
I20240503 14:12:18.855257 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1adea7aa88d4460dbc66959b5016f30b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.856001 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1adea7aa88d4460dbc66959b5016f30b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:18.856436 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f44e316ad8546aebf01ffcb826c3d46 has no consensus state
I20240503 14:12:18.859514 28661 log.cc:1196] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6371e68ee85a47cdb0e8e21f7124d94a
I20240503 14:12:18.860944 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_867e9a6baa014dada3c5c1ff572ab941" } new_table_name: "default.Table_a13cfbd07ce348578e9883c1205e8b9b" modify_external_catalogs: true
I20240503 14:12:18.861322 28661 ts_tablet_manager.cc:1909] T 6371e68ee85a47cdb0e8e21f7124d94a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:18.862167 28614 tablet_bootstrap.cc:492] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
W20240503 14:12:18.865775 28427 leader_election.cc:341] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:18.865947 28427 leader_election.cc:302] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.865808 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 642) took 2556 ms. Trace:
I20240503 14:12:18.866256 28661 rpcz_store.cc:276] 0503 14:12:16.309368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.309615 (+ 247us) service_pool.cc:225] Handling call
0503 14:12:18.749393 (+2439778us) ts_tablet_manager.cc:1120] Deleting tablet 6371e68ee85a47cdb0e8e21f7124d94a
0503 14:12:18.749491 (+ 98us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.790850 (+ 41359us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.852643 (+ 61793us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.865774 (+ 13131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2439547}
I20240503 14:12:18.866633 28614 ts_tablet_manager.cc:1367] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.000s
I20240503 14:12:18.866755 29701 raft_consensus.cc:2726] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.867005 29701 consensus_queue.cc:1035] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.867727 28661 tablet_replica.cc:330] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:18.867738 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6371e68ee85a47cdb0e8e21f7124d94a (table default.Table_355c3b14ae4f480dba47c2c90dc9dba2 [id=d7eac407621f4b03b21439fc2b0f7dc3]) successfully deleted
I20240503 14:12:18.868674 28614 raft_consensus.cc:348] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.868067 28661 raft_consensus.cc:2229] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.869681 28661 raft_consensus.cc:2258] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.876765 28661 ts_tablet_manager.cc:1875] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.877043 28614 raft_consensus.cc:374] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.877218 28614 raft_consensus.cc:729] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.877583 28614 consensus_queue.cc:260] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.880733 28600 tablet_bootstrap.cc:492] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:18.881108 28614 ts_tablet_manager.cc:1398] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:12:18.885479 28614 tablet_bootstrap.cc:492] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:18.894102 28603 ts_tablet_manager.cc:1888] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:18.897313 29750 consensus_queue.cc:1035] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:18.899067 28603 log.cc:1196] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/b7d4e4a1f0b14406be957426a45bf45d
I20240503 14:12:18.903439 28600 ts_tablet_manager.cc:1367] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.213s user 0.004s sys 0.002s
I20240503 14:12:18.903800 28603 ts_tablet_manager.cc:1909] T b7d4e4a1f0b14406be957426a45bf45d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:18.895730 28653 ts_tablet_manager.cc:1888] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.904475 28653 log.cc:1196] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/140de4d395d54b7696062ed96b8aa89c
I20240503 14:12:18.905279 28653 ts_tablet_manager.cc:1909] T 140de4d395d54b7696062ed96b8aa89c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:18.905100 28614 tablet_bootstrap.cc:654] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.906534 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet b7d4e4a1f0b14406be957426a45bf45d (table default.Table_d3937b734026471d8da300845379e2dd [id=27afd28af9ee45a9ad0aad92898ce78c]) successfully deleted
I20240503 14:12:18.908146 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 691) took 1073 ms. Trace:
I20240503 14:12:18.861897 28604 tablet_bootstrap.cc:654] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:18.910835 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e69e01bdbae349d0a61a295b6f6352f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e69e01bdbae349d0a61a295b6f6352f0 already in progress: creating tablet
I20240503 14:12:18.914769 28661 ts_tablet_manager.cc:1888] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:18.915256 29701 raft_consensus.cc:482] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:18.915257 28661 log.cc:1196] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a7295c2d3bd34e20a69986aadc9c7017
I20240503 14:12:18.915421 29701 raft_consensus.cc:504] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.915673 28661 ts_tablet_manager.cc:1909] T a7295c2d3bd34e20a69986aadc9c7017 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:18.916146 29701 leader_election.cc:288] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:18.916652 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 652) took 2509 ms. Trace:
I20240503 14:12:18.917243 28661 rpcz_store.cc:276] 0503 14:12:16.407366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.409039 (+ 1673us) service_pool.cc:225] Handling call
0503 14:12:18.867650 (+2458611us) ts_tablet_manager.cc:1120] Deleting tablet a7295c2d3bd34e20a69986aadc9c7017
0503 14:12:18.867665 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.883912 (+ 16247us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.914628 (+ 30716us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.916627 (+ 1999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457629}
I20240503 14:12:18.917956 28661 tablet_replica.cc:330] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:18.919348 28661 raft_consensus.cc:2229] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.920269 28661 raft_consensus.cc:2258] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.920163 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_920c4ca573c64022b064774de3fea713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:18.920970 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_920c4ca573c64022b064774de3fea713 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.921139 28614 tablet_bootstrap.cc:492] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:18.921293 28614 ts_tablet_manager.cc:1367] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.002s
I20240503 14:12:18.922749 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3c5bac1089fd4f629005b1381e2bca47" } modify_external_catalogs: true
I20240503 14:12:18.905792 28600 raft_consensus.cc:348] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.923029 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3c5bac1089fd4f629005b1381e2bca47" } modify_external_catalogs: true
I20240503 14:12:18.923139 28600 raft_consensus.cc:374] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.923302 28600 raft_consensus.cc:729] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.923651 28600 consensus_queue.cc:260] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.924096 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_087f6ee665a14f2a9f3cca94748dd55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:18.924539 28351 raft_consensus.cc:2454] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:18.924722 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_087f6ee665a14f2a9f3cca94748dd55b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:18.929558 28600 ts_tablet_manager.cc:1398] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 14:12:18.931454 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a7295c2d3bd34e20a69986aadc9c7017 (table default.Table_2e3372d35024401296239c262dc5fc5b [id=9e37697ebba24976a5f031ba8e9d87e5]) successfully deleted
I20240503 14:12:18.931854 28600 tablet_bootstrap.cc:492] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:18.934428 28600 tablet_bootstrap.cc:654] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.921975 28614 raft_consensus.cc:348] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.947474 28600 tablet_bootstrap.cc:492] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:18.948551 28600 ts_tablet_manager.cc:1367] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20240503 14:12:18.948951 28614 raft_consensus.cc:374] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.949054 28614 raft_consensus.cc:729] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.949404 28614 consensus_queue.cc:260] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.950004 28600 raft_consensus.cc:348] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.950459 28600 raft_consensus.cc:374] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.950603 28600 raft_consensus.cc:729] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.950786 28614 ts_tablet_manager.cc:1398] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240503 14:12:18.950955 28600 consensus_queue.cc:260] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:18.951645 28600 ts_tablet_manager.cc:1398] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:18.921005 28653 rpcz_store.cc:276] 0503 14:12:17.835122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.835210 (+ 88us) service_pool.cc:225] Handling call
0503 14:12:18.828715 (+993505us) ts_tablet_manager.cc:1120] Deleting tablet 140de4d395d54b7696062ed96b8aa89c
0503 14:12:18.828724 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:18.861125 (+ 32401us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.895559 (+ 34434us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:18.908067 (+ 12508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":993110}
I20240503 14:12:18.954030 28614 tablet_bootstrap.cc:492] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:18.934538 28427 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 5775006e9f8f4612aa70dfb316de9ea1
W20240503 14:12:18.955765 28427 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:18.955922 28427 leader_election.cc:302] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:18.956672 29700 raft_consensus.cc:2726] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:18.957239 28653 tablet_replica.cc:330] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:18.957504 28653 raft_consensus.cc:2229] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:18.957732 28653 raft_consensus.cc:2258] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:18.958423 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 140de4d395d54b7696062ed96b8aa89c (table default.Table_dbb3e0a033c84560a6458571881b25e5 [id=238df3dccaa54f4f956476ccb58b0816]) successfully deleted
I20240503 14:12:18.962895 28614 tablet_bootstrap.cc:654] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.958449 28661 ts_tablet_manager.cc:1875] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.965585 28653 ts_tablet_manager.cc:1875] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:18.966677 28600 tablet_bootstrap.cc:492] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:18.986259 28604 tablet_bootstrap.cc:492] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:18.986524 28604 ts_tablet_manager.cc:1367] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.150s user 0.005s sys 0.000s
I20240503 14:12:18.987598 28604 raft_consensus.cc:348] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.987916 28604 raft_consensus.cc:374] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.988010 28604 raft_consensus.cc:729] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:18.990486 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 888636e9beca43edab96a8736751409e consensus state has no leader
I20240503 14:12:18.990991 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet c5297664a4d943c28adfb6ac4f061680 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:18.991715 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7667eac51a7246a595705267a0d758a2" } new_table_name: "default.Table_0b6a50f68bf843b89ec7bc183ef30897" modify_external_catalogs: true
I20240503 14:12:18.990492 28604 consensus_queue.cc:260] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:18.998582 28604 ts_tablet_manager.cc:1398] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:12:18.999063 28604 tablet_bootstrap.cc:492] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:19.001667 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:19.001935 28600 tablet_bootstrap.cc:654] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.001986 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:19.002231 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a3023785b11c4ad7b8757a715f0952c5 (table default.Table_facfcbde0b2740359615d4907998c7fa [id=750bb7e271c241d2bbf14bd91dbc42d0]). Expected version 1 got 0
I20240503 14:12:19.006989 29700 raft_consensus.cc:482] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.007182 29700 raft_consensus.cc:504] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.008004 29700 leader_election.cc:288] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:19.008248 28324 tablet_service.cc:1467] Processing CreateTablet for tablet c4b122081bd34391809a2c5746fad7e6 (DEFAULT_TABLE table=default.Table_eb6121fb3aa742a3a8aa70b1426d9777 [id=4387686ed640413aa8ac41dcad75af40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.008754 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b122081bd34391809a2c5746fad7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.009812 28604 tablet_bootstrap.cc:654] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.012421 28333 tablet_service.cc:1467] Processing CreateTablet for tablet a4654e5b18e14aab9e976088a0939680 (DEFAULT_TABLE table=default.Table_f6d907b2cd874cd98d61854bb4c623fc [id=a68bb678f52445c5a1012b434b71cd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.012859 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4654e5b18e14aab9e976088a0939680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.015933 28458 tablet_service.cc:1467] Processing CreateTablet for tablet c4b122081bd34391809a2c5746fad7e6 (DEFAULT_TABLE table=default.Table_eb6121fb3aa742a3a8aa70b1426d9777 [id=4387686ed640413aa8ac41dcad75af40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.016443 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b122081bd34391809a2c5746fad7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.017139 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2c7ddd58b18f4c80adef853298b2019e" } modify_external_catalogs: true
I20240503 14:12:19.017387 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2c7ddd58b18f4c80adef853298b2019e" } modify_external_catalogs: true
I20240503 14:12:19.019207 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_15571f0ea45241568811001dfead7fff" } modify_external_catalogs: true
I20240503 14:12:19.019438 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_15571f0ea45241568811001dfead7fff" } modify_external_catalogs: true
I20240503 14:12:19.021052 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c5dfac02d4dfe932996d2a37fe031" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:19.025792 28614 tablet_bootstrap.cc:492] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
W20240503 14:12:19.025733 28427 leader_election.cc:341] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.025972 28614 ts_tablet_manager.cc:1367] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.007s
I20240503 14:12:19.027593 28030 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet df63f75c157a4e928d4dbbda06cfa5ce (table default.Table_e9e796aab9294a76b536ba18bd97568c [id=d29fe68058a14e178e42a10043a91738]). Expected version 3 got 0
I20240503 14:12:19.029054 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6b19c8158c64491dad1fdcddefb6b2b0" } new_table_name: "default.Table_981c33c4c24e4e848a74bde5c0a875d7" modify_external_catalogs: true
I20240503 14:12:19.029085 28614 raft_consensus.cc:348] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.029312 28614 raft_consensus.cc:374] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.029376 28614 raft_consensus.cc:729] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.029614 28614 consensus_queue.cc:260] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.030332 28614 ts_tablet_manager.cc:1398] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:19.037931 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 95a7da6f939b432da3986e000406a3ad (DEFAULT_TABLE table=default.Table_4db31dc8ab934c1595422943e50fe9c8 [id=b1f60dabcd014ea0a1c2357aed0ff7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.038470 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a7da6f939b432da3986e000406a3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.038764 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 755c5e04e0fb4acb9cf41f039ff4597d (DEFAULT_TABLE table=default.Table_12c636b581474b7998548e4a7d83e95c [id=c72d063cc38b4719aa886b1a14797cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.039175 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755c5e04e0fb4acb9cf41f039ff4597d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.040395 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 68c6bb1ce651489c9fcb6b839832d432 (DEFAULT_TABLE table=default.Table_caa773d21464476e8a52534f15c92525 [id=037b8b520f2e43c1a6a07b04ea56a035]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.040870 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c6bb1ce651489c9fcb6b839832d432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.041987 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a
I20240503 14:12:19.044543 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 1f44e316ad8546aebf01ffcb826c3d46 (DEFAULT_TABLE table=default.Table_1d16943aa7e24a50a783b77a82fdcc62 [id=d44c5f8eefaa434cbf37d2bc3e0fb4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.045046 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f44e316ad8546aebf01ffcb826c3d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.045035 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 68c6bb1ce651489c9fcb6b839832d432 (DEFAULT_TABLE table=default.Table_caa773d21464476e8a52534f15c92525 [id=037b8b520f2e43c1a6a07b04ea56a035]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.045261 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet cc004243d5a947e3a8852605b0414c95
I20240503 14:12:19.045467 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c6bb1ce651489c9fcb6b839832d432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.044589 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 95a7da6f939b432da3986e000406a3ad (DEFAULT_TABLE table=default.Table_4db31dc8ab934c1595422943e50fe9c8 [id=b1f60dabcd014ea0a1c2357aed0ff7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.046005 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 08cc84cf703048148abfe03b7dee12ce from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:19.046095 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a7da6f939b432da3986e000406a3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.047569 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6809b59e87644316b9a450d6b670da2b
I20240503 14:12:19.048174 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 6809b59e87644316b9a450d6b670da2b deleted and replaced by tablet 5d8ddcdb3c764dc59e9291725046d839
I20240503 14:12:19.048631 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 80b17f203de742fe859390ece6b4ca3f (DEFAULT_TABLE table=default.Table_f2e867e587ff49c293df696d8dd39cf0 [id=f345edf48c3f4ab2b7f2a392ef22c181]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.048663 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2aeaeda9485f4eb1953af7e447a714ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.049111 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b17f203de742fe859390ece6b4ca3f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.049423 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aeaeda9485f4eb1953af7e447a714ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.051119 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet cace51f8e5ff49f596c94f3698fdf9ef
I20240503 14:12:19.052516 28469 tablet_service.cc:1467] Processing CreateTablet for tablet a4654e5b18e14aab9e976088a0939680 (DEFAULT_TABLE table=default.Table_f6d907b2cd874cd98d61854bb4c623fc [id=a68bb678f52445c5a1012b434b71cd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.053025 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4654e5b18e14aab9e976088a0939680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.054697 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e1abe3457b4248fb826e4c21037ceb83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.055456 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1abe3457b4248fb826e4c21037ceb83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.056763 28472 tablet_service.cc:1467] Processing CreateTablet for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 (DEFAULT_TABLE table=default.Table_032d8e14bf694d2fbcd3b37c7ef106f9 [id=c6983b4f02404d5b8ea9edca241f26cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.057260 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91ddbc250c44fbfbf4ffe83433d8dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.051995 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6bfb8beb36ba4f4292703a3b2224d553"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.057816 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bfb8beb36ba4f4292703a3b2224d553 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:19.059425 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84c943a93a814633b7f71a4cb55b5ee3 consensus state has no leader
I20240503 14:12:19.060505 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 1f44e316ad8546aebf01ffcb826c3d46 (DEFAULT_TABLE table=default.Table_1d16943aa7e24a50a783b77a82fdcc62 [id=d44c5f8eefaa434cbf37d2bc3e0fb4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.060988 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f44e316ad8546aebf01ffcb826c3d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.062167 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet cc004243d5a947e3a8852605b0414c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.064432 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 755c5e04e0fb4acb9cf41f039ff4597d (DEFAULT_TABLE table=default.Table_12c636b581474b7998548e4a7d83e95c [id=c72d063cc38b4719aa886b1a14797cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.065026 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755c5e04e0fb4acb9cf41f039ff4597d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.069203 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.069650 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet cace51f8e5ff49f596c94f3698fdf9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.069710 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_83e923042498440797a6b72edecd6277" } new_table_name: "default.Table_ee9f2d2bd2b54111963a23deaf110216" modify_external_catalogs: true
I20240503 14:12:19.070688 28336 tablet_service.cc:1467] Processing CreateTablet for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 (DEFAULT_TABLE table=default.Table_032d8e14bf694d2fbcd3b37c7ef106f9 [id=c6983b4f02404d5b8ea9edca241f26cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.071199 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91ddbc250c44fbfbf4ffe83433d8dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.064976 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 6809b59e87644316b9a450d6b670da2b with delete_type TABLET_DATA_DELETED (replaced by tablet 5d8ddcdb3c764dc59e9291725046d839) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.076853 28031 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_facfcbde0b2740359615d4907998c7fa [id=750bb7e271c241d2bbf14bd91dbc42d0] alter complete (version 1)
I20240503 14:12:19.077852 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b530238524784f869cd09190d81b9726"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.078694 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b530238524784f869cd09190d81b9726 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.080047 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet bf8d2474104a41a989cd7bd4480367f5
I20240503 14:12:19.080997 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f8c8d1a54b104cbb88a7030052a89be3" } new_table_name: "default.Table_928c3ae55b104d519b4ad071f8babaac" modify_external_catalogs: true
I20240503 14:12:19.090071 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet bf8d2474104a41a989cd7bd4480367f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.091779 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 80b17f203de742fe859390ece6b4ca3f (DEFAULT_TABLE table=default.Table_f2e867e587ff49c293df696d8dd39cf0 [id=f345edf48c3f4ab2b7f2a392ef22c181]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.092386 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b17f203de742fe859390ece6b4ca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.093751 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 239f14be29e54cd29bb5e96622aacd55
I20240503 14:12:19.096382 29595 raft_consensus.cc:482] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.096585 29595 raft_consensus.cc:504] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.097337 29595 leader_election.cc:288] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:19.098583 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:19.098944 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 0e680f189add4b5c96de4046321aad2d (table default.Table_2d7ecf8640f64d82908807b4c4d4fd7f [id=d1088a94cd0442c9bd9837bf9f0e1ff9]). Expected version 2 got 0
I20240503 14:12:19.099212 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:19.099553 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
I20240503 14:12:19.099859 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet f2b861dccd964092ae0f06668c9b58fe (table default.Table_e5219472726e4702be79f785858e5ded [id=aaeb9d44d9f04208a57d49b81068662d]). Expected version 1 got 0
I20240503 14:12:19.100391 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.101224 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet cc004243d5a947e3a8852605b0414c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.101905 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 6809b59e87644316b9a450d6b670da2b with delete_type TABLET_DATA_DELETED (replaced by tablet 5d8ddcdb3c764dc59e9291725046d839) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.102460 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet cace51f8e5ff49f596c94f3698fdf9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.103188 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 239f14be29e54cd29bb5e96622aacd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.103962 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet bf8d2474104a41a989cd7bd4480367f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.104738 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 239f14be29e54cd29bb5e96622aacd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.105404 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 7218cefb13ea4983b2111ed5fd0c997b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.105657 28603 tablet_replica.cc:330] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:19.105872 28603 raft_consensus.cc:2229] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.106032 28603 raft_consensus.cc:2258] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.106192 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2404807f8500437ababca53f6388b636" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:19.106426 28491 raft_consensus.cc:2454] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:19.106602 28603 ts_tablet_manager.cc:1875] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.106940 28290 leader_election.cc:302] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:19.107184 29595 raft_consensus.cc:2781] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:19.107254 29595 raft_consensus.cc:482] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:19.107313 29595 raft_consensus.cc:3037] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.110507 29595 raft_consensus.cc:504] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.111670 29595 leader_election.cc:288] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:19.112619 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a already in progress: creating tablet
W20240503 14:12:19.113337 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet cc004243d5a947e3a8852605b0414c95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc004243d5a947e3a8852605b0414c95 already in progress: creating tablet
W20240503 14:12:19.114053 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6809b59e87644316b9a450d6b670da2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6809b59e87644316b9a450d6b670da2b already in progress: creating tablet
W20240503 14:12:19.114601 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet cace51f8e5ff49f596c94f3698fdf9ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cace51f8e5ff49f596c94f3698fdf9ef already in progress: creating tablet
W20240503 14:12:19.115231 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet bf8d2474104a41a989cd7bd4480367f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8d2474104a41a989cd7bd4480367f5 already in progress: creating tablet
W20240503 14:12:19.115818 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 239f14be29e54cd29bb5e96622aacd55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 239f14be29e54cd29bb5e96622aacd55 already in progress: creating tablet
I20240503 14:12:19.120266 28603 ts_tablet_manager.cc:1888] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.131290 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:19.135959 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 0e680f189add4b5c96de4046321aad2d (table default.Table_2d7ecf8640f64d82908807b4c4d4fd7f [id=d1088a94cd0442c9bd9837bf9f0e1ff9]). Expected version 2 got 0
I20240503 14:12:19.136382 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 02dd4c914a4846bab3a00408247ec64a (table default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc]). Expected version 2 got 0
I20240503 14:12:19.136760 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:19.137120 28030 catalog_manager.cc:5484] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:19.137800 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet df63f75c157a4e928d4dbbda06cfa5ce (table default.Table_e9e796aab9294a76b536ba18bd97568c [id=d29fe68058a14e178e42a10043a91738]). Expected version 3 got 0
I20240503 14:12:19.138092 28030 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:19.146286 29701 raft_consensus.cc:482] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.146462 29701 raft_consensus.cc:504] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.147161 29701 leader_election.cc:288] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:19.152832 28427 consensus_peers.cc:485] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b25c852aa7aa4fbfafdacec43f5bc426. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:19.155953 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad (DEFAULT_TABLE table=default.Table_701c15f6af2d4365aa030b14bbd1d82a [id=957bf351cc3c4161b4ddc0dcab2f963d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.156467 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.156944 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 986b1c549d934d4c85ed1b4b636bff48 (DEFAULT_TABLE table=default.Table_54d6cac450a145a89afabfe0c688c70c [id=dcc18e04e4054c38ac5fab6af16247f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.161518 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b1c549d934d4c85ed1b4b636bff48. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.161944 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 55024.5us
I20240503 14:12:19.162724 28603 log.cc:1196] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7218cefb13ea4983b2111ed5fd0c997b
I20240503 14:12:19.162818 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2404807f8500437ababca53f6388b636" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:19.163015 28491 raft_consensus.cc:3037] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.163064 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad (DEFAULT_TABLE table=default.Table_701c15f6af2d4365aa030b14bbd1d82a [id=957bf351cc3c4161b4ddc0dcab2f963d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.163249 28603 ts_tablet_manager.cc:1909] T 7218cefb13ea4983b2111ed5fd0c997b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:19.163465 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.164535 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_61e0a4e7d7544862b3028e7b3eec5bed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.165180 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61e0a4e7d7544862b3028e7b3eec5bed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.165689 28491 raft_consensus.cc:2454] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:19.169206 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 986b1c549d934d4c85ed1b4b636bff48 (DEFAULT_TABLE table=default.Table_54d6cac450a145a89afabfe0c688c70c [id=dcc18e04e4054c38ac5fab6af16247f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.169418 28326 tablet_service.cc:1467] Processing CreateTablet for tablet c99c15b4d13647e98cae1375b7f13827 (DEFAULT_TABLE table=default.Table_3307e69a5d4f4765b6ffc39e9df66fa1 [id=0364bf77a9e64e75b742ae75bea9684d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.169652 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b1c549d934d4c85ed1b4b636bff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.169927 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c15b4d13647e98cae1375b7f13827. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.177810 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet a3d1314841574fc8bcf3ccb16755e421 (table default.Table_ddd5f026bb204de99edef3cc54bd8347 [id=e5c443cd73d340e99806c89d5b8e659f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a3d1314841574fc8bcf3ccb16755e421 consensus state has no leader
W20240503 14:12:19.179626 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b46cfed6c8e42a6acf4d1fb128a8656 consensus state has no leader
I20240503 14:12:19.179745 29595 raft_consensus.cc:482] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.179919 29595 raft_consensus.cc:504] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.180202 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6aea0eb4f439481da1da149c63f49de8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.180738 29595 leader_election.cc:288] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:19.180934 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aea0eb4f439481da1da149c63f49de8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.190917 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet daf5d153ab5142c0902cc6e96629f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.191510 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f89fca0480d4bf28160384ae0421bce with delete_type TABLET_DATA_DELETED (replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.191910 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.192252 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 2404807f8500437ababca53f6388b636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.192833 28323 tablet_service.cc:1467] Processing CreateTablet for tablet f37ea39f88b04f0abb0445d4342406c3 (DEFAULT_TABLE table=default.Table_4a597910101d4133a1ad5201ce76c839 [id=8036b7900cfa4c06936ba3d3d38af612]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.193356 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37ea39f88b04f0abb0445d4342406c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.188947 28661 ts_tablet_manager.cc:1888] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:12:19.200438 28614 tablet_bootstrap.cc:492] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:19.199731 28661 log.cc:1196] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/dc337647c7cd402e91720478380f4696
I20240503 14:12:19.202701 28614 tablet_bootstrap.cc:654] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.203521 28603 tablet_replica.cc:330] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:19.203744 28603 raft_consensus.cc:2229] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.203907 28603 raft_consensus.cc:2258] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.204759 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1db04ab92c4af0847730a47635a697 (DEFAULT_TABLE table=default.Table_3d53a12202f9477788f519d5ef0c9531 [id=50aaea9c426b48e4bd44a475a481088f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.205230 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1db04ab92c4af0847730a47635a697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.205654 28469 tablet_service.cc:1467] Processing CreateTablet for tablet c99c15b4d13647e98cae1375b7f13827 (DEFAULT_TABLE table=default.Table_3307e69a5d4f4765b6ffc39e9df66fa1 [id=0364bf77a9e64e75b742ae75bea9684d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.206027 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c15b4d13647e98cae1375b7f13827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.208454 28462 tablet_service.cc:1467] Processing CreateTablet for tablet c725628f72884086b523b56ecf876069 (DEFAULT_TABLE table=default.Table_75f5b842e7054f3787e97d6641a08485 [id=ff7dde559e834a75a5e304762933375f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.208881 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725628f72884086b523b56ecf876069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.211226 28461 tablet_service.cc:1467] Processing CreateTablet for tablet f37ea39f88b04f0abb0445d4342406c3 (DEFAULT_TABLE table=default.Table_4a597910101d4133a1ad5201ce76c839 [id=8036b7900cfa4c06936ba3d3d38af612]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.211838 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37ea39f88b04f0abb0445d4342406c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.216850 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5e3649c71f4d4da710c69f85a82c25" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:19.217871 28336 tablet_service.cc:1467] Processing CreateTablet for tablet c725628f72884086b523b56ecf876069 (DEFAULT_TABLE table=default.Table_75f5b842e7054f3787e97d6641a08485 [id=ff7dde559e834a75a5e304762933375f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.220773 28661 ts_tablet_manager.cc:1909] T dc337647c7cd402e91720478380f4696 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:19.222939 28290 leader_election.cc:302] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:19.223587 29598 raft_consensus.cc:2781] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:19.223733 29598 raft_consensus.cc:686] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:19.223982 29598 consensus_queue.cc:237] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.224052 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf5d153ab5142c0902cc6e96629f412" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:19.224334 28491 raft_consensus.cc:2454] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:19.226668 28326 tablet_service.cc:1467] Processing CreateTablet for tablet f154dfaeb4fe47238ad928c8fc3521da (DEFAULT_TABLE table=default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.227272 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f154dfaeb4fe47238ad928c8fc3521da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.230479 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7218cefb13ea4983b2111ed5fd0c997b (table default.Table_e3c825867cf84b72bfb708fcc39f6acb [id=3a24797cc53b439da44cc77025fa9f92]) successfully deleted
I20240503 14:12:19.231947 28603 ts_tablet_manager.cc:1875] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.236335 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1db04ab92c4af0847730a47635a697 (DEFAULT_TABLE table=default.Table_3d53a12202f9477788f519d5ef0c9531 [id=50aaea9c426b48e4bd44a475a481088f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.236563 28194 tablet_service.cc:1467] Processing CreateTablet for tablet c4b122081bd34391809a2c5746fad7e6 (DEFAULT_TABLE table=default.Table_eb6121fb3aa742a3a8aa70b1426d9777 [id=4387686ed640413aa8ac41dcad75af40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.236922 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1db04ab92c4af0847730a47635a697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.239195 28189 tablet_service.cc:1467] Processing CreateTablet for tablet a4654e5b18e14aab9e976088a0939680 (DEFAULT_TABLE table=default.Table_f6d907b2cd874cd98d61854bb4c623fc [id=a68bb678f52445c5a1012b434b71cd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.242623 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b122081bd34391809a2c5746fad7e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.242810 28427 leader_election.cc:341] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.240471 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c5dfac02d4dfe932996d2a37fe031" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:19.246204 28180 tablet_service.cc:1467] Processing CreateTablet for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 (DEFAULT_TABLE table=default.Table_032d8e14bf694d2fbcd3b37c7ef106f9 [id=c6983b4f02404d5b8ea9edca241f26cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.249752 28473 tablet_service.cc:1467] Processing CreateTablet for tablet f154dfaeb4fe47238ad928c8fc3521da (DEFAULT_TABLE table=default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.250294 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f154dfaeb4fe47238ad928c8fc3521da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.251075 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 95a7da6f939b432da3986e000406a3ad (DEFAULT_TABLE table=default.Table_4db31dc8ab934c1595422943e50fe9c8 [id=b1f60dabcd014ea0a1c2357aed0ff7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.251546 28215 raft_consensus.cc:2454] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:19.252724 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.255331 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 239f14be29e54cd29bb5e96622aacd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.255966 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 68c6bb1ce651489c9fcb6b839832d432 (DEFAULT_TABLE table=default.Table_caa773d21464476e8a52534f15c92525 [id=037b8b520f2e43c1a6a07b04ea56a035]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.256536 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet cace51f8e5ff49f596c94f3698fdf9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.256978 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet cc004243d5a947e3a8852605b0414c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.257153 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 6809b59e87644316b9a450d6b670da2b with delete_type TABLET_DATA_DELETED (replaced by tablet 5d8ddcdb3c764dc59e9291725046d839) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.257275 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 755c5e04e0fb4acb9cf41f039ff4597d (DEFAULT_TABLE table=default.Table_12c636b581474b7998548e4a7d83e95c [id=c72d063cc38b4719aa886b1a14797cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.257655 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 80b17f203de742fe859390ece6b4ca3f (DEFAULT_TABLE table=default.Table_f2e867e587ff49c293df696d8dd39cf0 [id=f345edf48c3f4ab2b7f2a392ef22c181]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.257977 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 1f44e316ad8546aebf01ffcb826c3d46 (DEFAULT_TABLE table=default.Table_1d16943aa7e24a50a783b77a82fdcc62 [id=d44c5f8eefaa434cbf37d2bc3e0fb4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.259212 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet bf8d2474104a41a989cd7bd4480367f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.264313 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4654e5b18e14aab9e976088a0939680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.264827 29761 tablet.cc:1722] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:19.266314 29735 tablet.cc:1722] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:19.266731 28290 leader_election.cc:341] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2404807f8500437ababca53f6388b636
I20240503 14:12:19.267357 29595 consensus_queue.cc:1035] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:19.272751 28290 leader_election.cc:341] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2404807f8500437ababca53f6388b636
I20240503 14:12:19.272893 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1db04ab92c4af0847730a47635a697 (DEFAULT_TABLE table=default.Table_3d53a12202f9477788f519d5ef0c9531 [id=50aaea9c426b48e4bd44a475a481088f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:19.273298 28290 leader_election.cc:341] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: daf5d153ab5142c0902cc6e96629f412
I20240503 14:12:19.274597 29602 raft_consensus.cc:482] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.274870 29602 raft_consensus.cc:504] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.275022 28290 leader_election.cc:302] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:19.275656 29602 leader_election.cc:288] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:19.275225 28290 raft_consensus.cc:2670] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:19.276679 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 986b1c549d934d4c85ed1b4b636bff48 (DEFAULT_TABLE table=default.Table_54d6cac450a145a89afabfe0c688c70c [id=dcc18e04e4054c38ac5fab6af16247f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.277346 28182 tablet_service.cc:1467] Processing CreateTablet for tablet c99c15b4d13647e98cae1375b7f13827 (DEFAULT_TABLE table=default.Table_3307e69a5d4f4765b6ffc39e9df66fa1 [id=0364bf77a9e64e75b742ae75bea9684d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.277408 29595 consensus_queue.cc:1035] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:19.277974 28196 tablet_service.cc:1467] Processing CreateTablet for tablet f37ea39f88b04f0abb0445d4342406c3 (DEFAULT_TABLE table=default.Table_4a597910101d4133a1ad5201ce76c839 [id=8036b7900cfa4c06936ba3d3d38af612]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.278729 28191 tablet_service.cc:1467] Processing CreateTablet for tablet f154dfaeb4fe47238ad928c8fc3521da (DEFAULT_TABLE table=default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.279278 28197 tablet_service.cc:1467] Processing CreateTablet for tablet c725628f72884086b523b56ecf876069 (DEFAULT_TABLE table=default.Table_75f5b842e7054f3787e97d6641a08485 [id=ff7dde559e834a75a5e304762933375f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.280841 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495190fca06a40a386c8215eb2195920" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:19.281759 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495190fca06a40a386c8215eb2195920" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:19.281893 28150 leader_election.cc:341] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:19.282387 28150 leader_election.cc:341] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.282516 28150 leader_election.cc:302] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:19.295207 28604 tablet_bootstrap.cc:492] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:19.296486 28603 ts_tablet_manager.cc:1888] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.296779 28603 log.cc:1196] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/daf5d153ab5142c0902cc6e96629f412
I20240503 14:12:19.296638 28604 ts_tablet_manager.cc:1367] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.298s user 0.109s sys 0.109s
I20240503 14:12:19.297322 28603 ts_tablet_manager.cc:1909] T daf5d153ab5142c0902cc6e96629f412 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:19.297924 28603 tablet_replica.cc:330] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:19.298538 28603 raft_consensus.cc:2229] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.298828 28603 raft_consensus.cc:2258] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.300698 28603 ts_tablet_manager.cc:1875] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.306067 28604 raft_consensus.cc:348] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.307993 28604 raft_consensus.cc:374] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.308890 28604 raft_consensus.cc:729] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.311308 28604 consensus_queue.cc:260] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.317741 28604 ts_tablet_manager.cc:1398] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.020s user 0.003s sys 0.001s
I20240503 14:12:19.319334 28604 tablet_bootstrap.cc:492] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:19.322741 28604 tablet_bootstrap.cc:654] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.328684 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:19.334545 28604 tablet_bootstrap.cc:492] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:19.334951 28604 ts_tablet_manager.cc:1367] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20240503 14:12:19.336728 28604 raft_consensus.cc:348] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.337186 28604 raft_consensus.cc:374] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.337391 28604 raft_consensus.cc:729] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.337873 28604 consensus_queue.cc:260] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.339311 28604 ts_tablet_manager.cc:1398] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:19.339900 28604 tablet_bootstrap.cc:492] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:19.343251 28604 tablet_bootstrap.cc:654] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.345005 28603 ts_tablet_manager.cc:1888] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.348223 28603 log.cc:1196] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/2f89fca0480d4bf28160384ae0421bce
I20240503 14:12:19.348776 28603 ts_tablet_manager.cc:1909] T 2f89fca0480d4bf28160384ae0421bce P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:19.352954 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5e3649c71f4d4da710c69f85a82c25" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:19.353386 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755c5e04e0fb4acb9cf41f039ff4597d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.353736 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1db04ab92c4af0847730a47635a697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.353777 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad (DEFAULT_TABLE table=default.Table_701c15f6af2d4365aa030b14bbd1d82a [id=957bf351cc3c4161b4ddc0dcab2f963d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.354269 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da9c6270e0c4de8a4cb9d68e2dc69ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.355577 28603 tablet_replica.cc:330] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:19.355810 28603 raft_consensus.cc:2229] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.355998 28603 raft_consensus.cc:2258] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.359977 28603 ts_tablet_manager.cc:1875] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.360308 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 08cc84cf703048148abfe03b7dee12ce
I20240503 14:12:19.360623 28604 tablet_bootstrap.cc:492] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:19.360961 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 08cc84cf703048148abfe03b7dee12ce deleted and replaced by tablet 5d308130f0dc4c16ac73058db332b41d
I20240503 14:12:19.360872 28604 ts_tablet_manager.cc:1367] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.007s
I20240503 14:12:19.362042 28604 raft_consensus.cc:348] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.362103 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f760a9cbcc20443ba5c28ec36efb98db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.362486 28604 raft_consensus.cc:374] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.362612 28604 raft_consensus.cc:729] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:19.362848 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f760a9cbcc20443ba5c28ec36efb98db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.362917 28604 consensus_queue.cc:260] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.365072 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_50e8eea84f564a4da088dd0efa136f69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.365746 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50e8eea84f564a4da088dd0efa136f69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.366775 28603 ts_tablet_manager.cc:1888] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.366904 28603 log.cc:1196] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:19.367192 28603 ts_tablet_manager.cc:1909] T e0a92fc3833e49aaab8c5415de9bb685 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:19.367255 28604 ts_tablet_manager.cc:1398] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:19.367594 28604 tablet_bootstrap.cc:492] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:19.368033 28603 tablet_replica.cc:330] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:19.369344 28604 tablet_bootstrap.cc:654] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.371601 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f7bbe9e54e404d698feb87bac95370d0" } new_table_name: "default.Table_d6bd120570ec4a8ab95342666e7d1d83" modify_external_catalogs: true
I20240503 14:12:19.374017 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a7da6f939b432da3986e000406a3ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.374138 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f7b805aee554f8fa4c198ce3409c6e6 consensus state has no leader
I20240503 14:12:19.374888 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f154dfaeb4fe47238ad928c8fc3521da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.374720 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cc3d602d565c4c32b9127717dc19a658"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.375425 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc3d602d565c4c32b9127717dc19a658 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.376361 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c6bb1ce651489c9fcb6b839832d432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.377094 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f44e316ad8546aebf01ffcb826c3d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.381927 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet daf5d153ab5142c0902cc6e96629f412 (table default.Table_6673b8d0a18d42c3a18f6e611c596580 [id=3c43a0fcf51143179a1ed0e1b8e333c1]) successfully deleted
I20240503 14:12:19.382449 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 2f89fca0480d4bf28160384ae0421bce (table default.Table_777d8101e5ba4707a324b1ea2b061d2a [id=9a6689ece91646069be1867e8f141392]) successfully deleted
I20240503 14:12:19.382871 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e0a92fc3833e49aaab8c5415de9bb685 (table default.Table_9953322d44f34ce5ad1123ba2784b66f [id=5ae48705fb644bddba8e69d66c640a68]) successfully deleted
I20240503 14:12:19.389434 28427 leader_election.cc:302] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:19.390084 29750 raft_consensus.cc:2781] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:19.390213 29750 raft_consensus.cc:482] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:19.390276 29750 raft_consensus.cc:3037] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.393968 29750 raft_consensus.cc:504] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.395552 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c5dfac02d4dfe932996d2a37fe031" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:19.396415 28427 leader_election.cc:341] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.397090 29750 leader_election.cc:288] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:19.398815 28427 consensus_peers.cc:485] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b25c852aa7aa4fbfafdacec43f5bc426. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.400501 28427 leader_election.cc:341] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.400674 28427 leader_election.cc:302] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:19.401213 29700 raft_consensus.cc:2726] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:19.409085 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cc84cf703048148abfe03b7dee12ce with delete_type TABLET_DATA_DELETED (replaced by tablet 5d308130f0dc4c16ac73058db332b41d) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.410856 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cc84cf703048148abfe03b7dee12ce with delete_type TABLET_DATA_DELETED (replaced by tablet 5d308130f0dc4c16ac73058db332b41d) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.411446 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91ddbc250c44fbfbf4ffe83433d8dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.416411 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b17f203de742fe859390ece6b4ca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.420349 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725628f72884086b523b56ecf876069. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.426203 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 888c28447db64751adca55ae769d0a37 consensus state has no leader
I20240503 14:12:19.426272 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_06dc28f66ee6455e8cea52e834c0442c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.426653 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e4311f501e744a4a859e4eef2878127c" } modify_external_catalogs: true
I20240503 14:12:19.426883 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e4311f501e744a4a859e4eef2878127c" } modify_external_catalogs: true
W20240503 14:12:19.427059 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06dc28f66ee6455e8cea52e834c0442c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.427184 28653 ts_tablet_manager.cc:1888] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.427326 28653 log.cc:1196] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7abcb8658fc34c939aa74cf0bd981ae4
I20240503 14:12:19.427657 28653 ts_tablet_manager.cc:1909] T 7abcb8658fc34c939aa74cf0bd981ae4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:19.429967 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3d53a12202f9477788f519d5ef0c9531" } modify_external_catalogs: true
I20240503 14:12:19.430181 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3d53a12202f9477788f519d5ef0c9531" } modify_external_catalogs: true
I20240503 14:12:19.432471 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
I20240503 14:12:19.432760 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet e6c90869b5034128864faaf05697c230 (table default.Table_5dcdff74f1f7407084a45dc9fe588103 [id=47ade8624b00424d8fc04dbca5174ff4]). Expected version 3 got 0
I20240503 14:12:19.432951 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0e680f189add4b5c96de4046321aad2d (table default.Table_2d7ecf8640f64d82908807b4c4d4fd7f [id=d1088a94cd0442c9bd9837bf9f0e1ff9]). Expected version 2 got 0
I20240503 14:12:19.433125 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]). Expected version 1 got 0
I20240503 14:12:19.433650 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b1c549d934d4c85ed1b4b636bff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.435204 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_573a1680a2fa4db793e25023dd9eea51" } modify_external_catalogs: true
I20240503 14:12:19.436808 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_573a1680a2fa4db793e25023dd9eea51" } modify_external_catalogs: true
I20240503 14:12:19.441217 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c15b4d13647e98cae1375b7f13827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.446262 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37ea39f88b04f0abb0445d4342406c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.448778 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d45788ae644a8581acca54cc55dd3f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.455914 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]). Expected version 3 got 0
I20240503 14:12:19.456818 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2d7ecf8640f64d82908807b4c4d4fd7f" } new_table_name: "default.Table_d73c3c2812414afc8bab352a368604cf" modify_external_catalogs: true
I20240503 14:12:19.457602 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725628f72884086b523b56ecf876069. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.458891 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a" } modify_external_catalogs: true
I20240503 14:12:19.459210 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a" } modify_external_catalogs: true
W20240503 14:12:19.460566 28427 consensus_peers.cc:485] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00b52ea336524beea2f255c8720f7472. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:19.456169 29750 raft_consensus.cc:482] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.460970 29750 raft_consensus.cc:504] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.461148 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_db04b673d60644559c17717f39eab7e2" } modify_external_catalogs: true
I20240503 14:12:19.461392 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_db04b673d60644559c17717f39eab7e2" } modify_external_catalogs: true
I20240503 14:12:19.461822 29750 leader_election.cc:288] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:19.462736 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dd4c914a4846bab3a00408247ec64a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:19.462922 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b7ca19dd15f48a68ace7fefc7ae660f consensus state has no leader
I20240503 14:12:19.463304 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_46e0d74510e14187b44fee0f210792be" } new_table_name: "default.Table_d9f3904b328a47168ecb632d99de352b" modify_external_catalogs: true
W20240503 14:12:19.463431 28427 leader_election.cc:341] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.466308 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 08cc84cf703048148abfe03b7dee12ce from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:19.468544 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
I20240503 14:12:19.468847 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 240abf8c2b2343408fc98a8dac1fdf09 (table default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692]). Expected version 2 got 0
I20240503 14:12:19.461175 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 651) took 3054 ms. Trace:
I20240503 14:12:19.473273 28661 rpcz_store.cc:276] 0503 14:12:16.407185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.413197 (+ 6012us) service_pool.cc:225] Handling call
0503 14:12:18.917896 (+2504699us) ts_tablet_manager.cc:1120] Deleting tablet dc337647c7cd402e91720478380f4696
0503 14:12:18.917906 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.148297 (+230391us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.187012 (+ 38715us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.461150 (+274138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":321,"tablet-delete.queue_time_us":2504445}
I20240503 14:12:19.474027 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet dc337647c7cd402e91720478380f4696 (table default.Table_573a1680a2fa4db793e25023dd9eea51 [id=acb43ab939ff46428a795f618c6750dd]) successfully deleted
I20240503 14:12:19.479748 28600 tablet_bootstrap.cc:492] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:19.479964 28600 ts_tablet_manager.cc:1367] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.513s user 0.007s sys 0.006s
I20240503 14:12:19.481645 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2c1db04ab92c4af0847730a47635a697
I20240503 14:12:19.484601 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1db04ab92c4af0847730a47635a697 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.490378 28604 tablet_bootstrap.cc:492] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:19.490658 28604 ts_tablet_manager.cc:1367] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.123s user 0.035s sys 0.055s
I20240503 14:12:19.491551 28604 raft_consensus.cc:348] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.491856 28604 raft_consensus.cc:374] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.491981 28604 raft_consensus.cc:729] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.492360 28604 consensus_queue.cc:260] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.499958 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 37364d5facc241f4bbccc979a50a0ca8
I20240503 14:12:19.502959 28600 raft_consensus.cc:348] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.503228 28600 raft_consensus.cc:374] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.503293 28600 raft_consensus.cc:729] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.503499 28600 consensus_queue.cc:260] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.504055 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1db04ab92c4af0847730a47635a697 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.504886 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 37364d5facc241f4bbccc979a50a0ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.507292 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 692) took 1670 ms. Trace:
I20240503 14:12:19.507735 28653 rpcz_store.cc:276] 0503 14:12:17.836428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.836486 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:18.957134 (+1120648us) ts_tablet_manager.cc:1120] Deleting tablet 7abcb8658fc34c939aa74cf0bd981ae4
0503 14:12:18.957144 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.249031 (+291887us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.427036 (+178005us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.507257 (+ 80221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56426,"tablet-delete.queue_time_us":1120242}
I20240503 14:12:19.515893 29602 raft_consensus.cc:2726] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:19.516100 29602 raft_consensus.cc:482] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.522634 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 03700c1efc08471cb67dfe3a8016a740
I20240503 14:12:19.523999 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 03700c1efc08471cb67dfe3a8016a740 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.529491 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 693) took 1675 ms. Trace:
I20240503 14:12:19.529647 28653 rpcz_store.cc:276] 0503 14:12:17.853611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.853685 (+ 74us) service_pool.cc:225] Handling call
0503 14:12:19.508101 (+1654416us) ts_tablet_manager.cc:1120] Deleting tablet e45df464f6954a56bec8b107b350320c
0503 14:12:19.508114 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.529447 (+ 21333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654099}
I20240503 14:12:19.492887 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 653) took 2988 ms. Trace:
I20240503 14:12:19.535496 28661 rpcz_store.cc:276] 0503 14:12:16.487580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.487656 (+ 76us) service_pool.cc:225] Handling call
0503 14:12:19.475497 (+2987841us) ts_tablet_manager.cc:1120] Deleting tablet 0c7779f7a9544e17aea7135eab2ccd5f
0503 14:12:19.475514 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.475659 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2987606}
I20240503 14:12:19.501020 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 37364d5facc241f4bbccc979a50a0ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.524340 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 03700c1efc08471cb67dfe3a8016a740 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.537705 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 655) took 3030 ms. Trace:
I20240503 14:12:19.537835 28661 rpcz_store.cc:276] 0503 14:12:16.507090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.507144 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:19.535849 (+3028705us) ts_tablet_manager.cc:1120] Deleting tablet af0137d7b1034a79bf4947c7d3f3fd57
0503 14:12:19.537516 (+ 1667us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.537676 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3028398}
W20240503 14:12:19.538637 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0c7779f7a9544e17aea7135eab2ccd5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c7779f7a9544e17aea7135eab2ccd5f already in progress: creating tablet
W20240503 14:12:19.539021 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet af0137d7b1034a79bf4947c7d3f3fd57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af0137d7b1034a79bf4947c7d3f3fd57 already in progress: creating tablet
I20240503 14:12:19.539980 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 656) took 3032 ms. Trace:
I20240503 14:12:19.540099 28661 rpcz_store.cc:276] 0503 14:12:16.507788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.507838 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:19.538110 (+3030272us) ts_tablet_manager.cc:1120] Deleting tablet 64f226db2fdb47ba9472a3aa65b953c3
0503 14:12:19.539827 (+ 1717us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.539960 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030047}
W20240503 14:12:19.540737 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 64f226db2fdb47ba9472a3aa65b953c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f226db2fdb47ba9472a3aa65b953c3 already in progress: creating tablet
I20240503 14:12:19.541030 29602 raft_consensus.cc:504] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.541231 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 694) took 1685 ms. Trace:
I20240503 14:12:19.539738 28604 ts_tablet_manager.cc:1398] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.049s user 0.000s sys 0.002s
I20240503 14:12:19.541805 29602 leader_election.cc:288] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:19.541873 28600 ts_tablet_manager.cc:1398] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.062s user 0.006s sys 0.006s
I20240503 14:12:19.543323 28614 tablet_bootstrap.cc:492] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:19.544108 28614 ts_tablet_manager.cc:1367] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.344s user 0.005s sys 0.009s
I20240503 14:12:19.545639 28600 tablet_bootstrap.cc:492] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:19.550158 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 654) took 3043 ms. Trace:
I20240503 14:12:19.550326 28661 rpcz_store.cc:276] 0503 14:12:16.506968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.508634 (+ 1666us) service_pool.cc:225] Handling call
0503 14:12:19.540378 (+3031744us) ts_tablet_manager.cc:1120] Deleting tablet 2b52ef7ecedf4273939406581d843980
0503 14:12:19.549968 (+ 9590us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.550126 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3031506}
I20240503 14:12:19.550812 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 658) took 2996 ms. Trace:
I20240503 14:12:19.550920 28661 rpcz_store.cc:276] 0503 14:12:16.554110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.557405 (+ 3295us) service_pool.cc:225] Handling call
0503 14:12:19.550671 (+2993266us) ts_tablet_manager.cc:1120] Deleting tablet f080c95ff5ca41e0b3d27382040107e9
0503 14:12:19.550684 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.550790 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2991805}
W20240503 14:12:19.556401 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2b52ef7ecedf4273939406581d843980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b52ef7ecedf4273939406581d843980 already in progress: creating tablet
W20240503 14:12:19.556828 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f080c95ff5ca41e0b3d27382040107e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f080c95ff5ca41e0b3d27382040107e9 already in progress: creating tablet
I20240503 14:12:19.560101 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 659) took 2907 ms. Trace:
I20240503 14:12:19.560226 28661 rpcz_store.cc:276] 0503 14:12:16.653069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.653160 (+ 91us) service_pool.cc:225] Handling call
0503 14:12:19.551172 (+2898012us) ts_tablet_manager.cc:1120] Deleting tablet 07354861ee6142d6b1fbe573397d3160
0503 14:12:19.559892 (+ 8720us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.560050 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2893718}
I20240503 14:12:19.561141 28603 raft_consensus.cc:2229] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:19.561709 28604 tablet_bootstrap.cc:492] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:19.563792 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 07354861ee6142d6b1fbe573397d3160 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07354861ee6142d6b1fbe573397d3160 already in progress: creating tablet
I20240503 14:12:19.563473 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 660) took 2875 ms. Trace:
I20240503 14:12:19.565382 28661 rpcz_store.cc:276] 0503 14:12:16.687844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.687939 (+ 95us) service_pool.cc:225] Handling call
0503 14:12:19.560520 (+2872581us) ts_tablet_manager.cc:1120] Deleting tablet 0f75e102dd4249c3800e1fcb4339acf3
0503 14:12:19.563321 (+ 2801us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.563450 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":2872315}
I20240503 14:12:19.565876 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 662) took 2877 ms. Trace:
I20240503 14:12:19.565963 28661 rpcz_store.cc:276] 0503 14:12:16.688604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.688652 (+ 48us) service_pool.cc:225] Handling call
0503 14:12:19.565724 (+2877072us) ts_tablet_manager.cc:1120] Deleting tablet 1292a339f00247369ebeaa35752d51f8
0503 14:12:19.565735 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.565850 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2876880}
I20240503 14:12:19.566709 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 661) took 2878 ms. Trace:
W20240503 14:12:19.566721 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0f75e102dd4249c3800e1fcb4339acf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f75e102dd4249c3800e1fcb4339acf3 already in progress: creating tablet
I20240503 14:12:19.566799 28661 rpcz_store.cc:276] 0503 14:12:16.688458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.689011 (+ 553us) service_pool.cc:225] Handling call
0503 14:12:19.566180 (+2877169us) ts_tablet_manager.cc:1120] Deleting tablet 62e4f6e3908e420e9fa156ee24b437d6
0503 14:12:19.566594 (+ 414us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.566690 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2876989}
W20240503 14:12:19.567101 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1292a339f00247369ebeaa35752d51f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1292a339f00247369ebeaa35752d51f8 already in progress: creating tablet
I20240503 14:12:19.567157 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 673) took 2651 ms. Trace:
I20240503 14:12:19.567222 28661 rpcz_store.cc:276] 0503 14:12:16.915931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.916007 (+ 76us) service_pool.cc:225] Handling call
0503 14:12:19.567038 (+2651031us) ts_tablet_manager.cc:1120] Deleting tablet f6061918660e4070a0c8339cb5009ff8
0503 14:12:19.567048 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.567131 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2650756}
I20240503 14:12:19.567466 28661 tablet_replica.cc:330] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:19.567664 28661 raft_consensus.cc:2229] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.567840 28661 raft_consensus.cc:2258] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.568246 28614 raft_consensus.cc:348] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.568460 28614 raft_consensus.cc:374] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.568516 28614 raft_consensus.cc:729] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.568768 28614 consensus_queue.cc:260] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.569355 28614 ts_tablet_manager.cc:1398] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 14:12:19.569744 28614 tablet_bootstrap.cc:492] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:19.569908 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e3b970f0246245259885e419a838a1a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.570617 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3b970f0246245259885e419a838a1a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.571146 28603 pending_rounds.cc:62] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:19.571216 28603 pending_rounds.cc:68] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597728664473600 op_type: NO_OP noop_request { }
I20240503 14:12:19.571341 28603 raft_consensus.cc:2866] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:19.571424 28603 raft_consensus.cc:2258] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.572063 28661 ts_tablet_manager.cc:1875] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.572911 29735 tablet.cc:1722] T 8f0795253db548fb84c2d7bb3dc48335 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:19.573616 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ae6805422d0d43b6acde2ee0ccb6a6d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.574317 29761 tablet.cc:1722] T 8f0795253db548fb84c2d7bb3dc48335 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 14:12:19.574501 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae6805422d0d43b6acde2ee0ccb6a6d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:19.577103 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 62e4f6e3908e420e9fa156ee24b437d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62e4f6e3908e420e9fa156ee24b437d6 already in progress: creating tablet
W20240503 14:12:19.577459 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f6061918660e4070a0c8339cb5009ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6061918660e4070a0c8339cb5009ff8 already in progress: creating tablet
I20240503 14:12:19.578619 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ced17a17c1af43d0be48f006c4b2a195"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.579401 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ced17a17c1af43d0be48f006c4b2a195 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.581202 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1c2668b8ddb741298f29976408a1caee" } new_table_name: "default.Table_e20c31514a2a4f089d76d0d26ece4265" modify_external_catalogs: true
I20240503 14:12:19.590932 28603 ts_tablet_manager.cc:1875] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.591050 28604 tablet_bootstrap.cc:654] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.593080 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6de9e159d2ba47039dd6bd207a9cddeb
I20240503 14:12:19.593890 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.594120 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9011de10a7ff497fb35ccb3ada7315ef
I20240503 14:12:19.594745 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 9011de10a7ff497fb35ccb3ada7315ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.595388 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.595808 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 9011de10a7ff497fb35ccb3ada7315ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.603117 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.604022 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.602449 28661 ts_tablet_manager.cc:1888] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:19.605177 28661 log.cc:1196] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0926e523ef454a2089f4d40a328c1c6d
I20240503 14:12:19.605576 28661 ts_tablet_manager.cc:1909] T 0926e523ef454a2089f4d40a328c1c6d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:19.605755 28603 ts_tablet_manager.cc:1888] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:19.606007 28603 log.cc:1196] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/2404807f8500437ababca53f6388b636
I20240503 14:12:19.606410 28603 ts_tablet_manager.cc:1909] T 2404807f8500437ababca53f6388b636 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:19.607277 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 2404807f8500437ababca53f6388b636 (table default.Table_7d8f2c48beb5472fb05f0cbd475f7182 [id=a396cc6875ba41d29e801308d3f274d0]) successfully deleted
W20240503 14:12:19.608256 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 08cc84cf703048148abfe03b7dee12ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cc84cf703048148abfe03b7dee12ce already in progress: creating tablet
I20240503 14:12:19.608695 28614 tablet_bootstrap.cc:654] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:19.608597 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2c1db04ab92c4af0847730a47635a697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c1db04ab92c4af0847730a47635a697 already in progress: creating tablet
I20240503 14:12:19.609304 28603 tablet_replica.cc:330] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:19.609542 28603 raft_consensus.cc:2229] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:19.609531 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 37364d5facc241f4bbccc979a50a0ca8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37364d5facc241f4bbccc979a50a0ca8 already in progress: creating tablet
W20240503 14:12:19.611079 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 03700c1efc08471cb67dfe3a8016a740 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03700c1efc08471cb67dfe3a8016a740 already in progress: creating tablet
W20240503 14:12:19.611342 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6de9e159d2ba47039dd6bd207a9cddeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6de9e159d2ba47039dd6bd207a9cddeb already in progress: creating tablet
I20240503 14:12:19.611891 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 2404807f8500437ababca53f6388b636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.612268 28603 raft_consensus.cc:2258] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.541342 28653 rpcz_store.cc:276] 0503 14:12:17.855592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.855660 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:19.529970 (+1674310us) ts_tablet_manager.cc:1120] Deleting tablet 5757e6e8703640519f7d38754cc0c8ce
0503 14:12:19.529984 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.541195 (+ 11211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1674016}
I20240503 14:12:19.612627 28653 tablet_replica.cc:330] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:19.612866 28653 raft_consensus.cc:2229] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.613034 28653 raft_consensus.cc:2258] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.613754 28603 ts_tablet_manager.cc:1875] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.615483 28653 ts_tablet_manager.cc:1875] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.616397 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 677) took 2502 ms. Trace:
I20240503 14:12:19.616499 28661 rpcz_store.cc:276] 0503 14:12:17.114910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.115169 (+ 259us) service_pool.cc:225] Handling call
0503 14:12:19.567424 (+2452255us) ts_tablet_manager.cc:1120] Deleting tablet 0926e523ef454a2089f4d40a328c1c6d
0503 14:12:19.567432 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.575678 (+ 8246us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.602324 (+ 26646us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.616372 (+ 14048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399006}
I20240503 14:12:19.617003 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0926e523ef454a2089f4d40a328c1c6d (table default.Table_193453f3dc0a440db9aaeada83e07e1a [id=3e3945bf21d24fedb9b811e4e3648567]) successfully deleted
I20240503 14:12:19.618021 28661 tablet_replica.cc:330] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:19.618257 28661 raft_consensus.cc:2229] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.618410 28661 raft_consensus.cc:2258] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.619006 28661 ts_tablet_manager.cc:1875] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.619469 28028 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet c924a74ee836406fb3157baef6f4c901 (table default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1]). Expected version 1 got 0
W20240503 14:12:19.620046 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 337642us
I20240503 14:12:19.620860 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:19.620862 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:19.621086 28489 raft_consensus.cc:2454] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:19.622978 28150 leader_election.cc:302] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:19.623337 28150 leader_election.cc:341] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.624541 29772 raft_consensus.cc:2781] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:19.624666 29772 raft_consensus.cc:482] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:19.624753 29772 raft_consensus.cc:3037] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.628046 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 0e680f189add4b5c96de4046321aad2d (table default.Table_d73c3c2812414afc8bab352a368604cf [id=d1088a94cd0442c9bd9837bf9f0e1ff9]). Expected version 3 got 2
I20240503 14:12:19.628451 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 65853212c2404597b015ddc8ea99ee47 (table default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420]). Expected version 2 got 0
I20240503 14:12:19.631633 28603 ts_tablet_manager.cc:1888] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:19.631754 28603 log.cc:1196] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/9011de10a7ff497fb35ccb3ada7315ef
I20240503 14:12:19.632942 28614 tablet_bootstrap.cc:492] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:19.633178 28614 ts_tablet_manager.cc:1367] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.000s
I20240503 14:12:19.633342 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c0420d9a80af4738972115df9798052c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.634131 28614 raft_consensus.cc:348] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.634335 28614 raft_consensus.cc:374] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.634382 28614 raft_consensus.cc:729] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.634598 28614 consensus_queue.cc:260] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.635109 28614 ts_tablet_manager.cc:1398] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:19.635402 28614 tablet_bootstrap.cc:492] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:19.639317 29772 raft_consensus.cc:504] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.640041 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2ef0e74fb73844b3a4d3b680c04d73e1" } new_table_name: "default.Table_c9222bb788244d4ebac69f05a6c32245" modify_external_catalogs: true
I20240503 14:12:19.641083 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:19.641310 28489 raft_consensus.cc:3037] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.642041 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:19.642830 28150 leader_election.cc:341] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.640394 29772 leader_election.cc:288] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:19.645092 28600 tablet_bootstrap.cc:654] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.646167 28489 raft_consensus.cc:2454] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:19.648411 28150 leader_election.cc:302] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:19.648759 29772 raft_consensus.cc:2781] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:19.648900 29772 raft_consensus.cc:686] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:19.649166 29772 consensus_queue.cc:237] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.650168 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 332ecc478e49424d9e7d7decbbc2c29c from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:19.650604 29771 raft_consensus.cc:482] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.650748 29771 raft_consensus.cc:504] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.651459 29771 leader_election.cc:288] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:19.652810 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0420d9a80af4738972115df9798052c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:19.652921 28150 leader_election.cc:341] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 2f89fca0480d4bf28160384ae0421bce
I20240503 14:12:19.655320 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f89fca0480d4bf28160384ae0421bce" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:19.659090 28614 tablet_bootstrap.cc:654] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:19.659329 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6615052ddbde487e840b6fd77a140e58 consensus state has no leader
W20240503 14:12:19.660295 28150 leader_election.cc:341] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.660440 28150 leader_election.cc:302] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:19.661010 29771 raft_consensus.cc:2726] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:19.669240 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
I20240503 14:12:19.671607 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_deb8a08c99c54d0cbff9bcb72144aba6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.672699 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deb8a08c99c54d0cbff9bcb72144aba6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.674439 28653 ts_tablet_manager.cc:1888] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.674577 28653 log.cc:1196] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/60206e8e73e84e409ed6aba222d9ff4b
I20240503 14:12:19.674402 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a65e77322444449cadf78698f2c98fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.675091 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a65e77322444449cadf78698f2c98fd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.679700 28603 ts_tablet_manager.cc:1909] T 9011de10a7ff497fb35ccb3ada7315ef P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:19.679637 29761 tablet.cc:1722] T 0e680f189add4b5c96de4046321aad2d P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:19.680692 29735 tablet.cc:1722] T 0e680f189add4b5c96de4046321aad2d P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:19.686352 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]) successfully deleted
I20240503 14:12:19.688165 29750 raft_consensus.cc:482] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.688441 29750 raft_consensus.cc:504] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.689230 29750 leader_election.cc:288] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:19.690538 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:19.692499 28427 leader_election.cc:341] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.695205 28600 tablet_bootstrap.cc:492] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:19.695379 28600 ts_tablet_manager.cc:1367] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.150s user 0.003s sys 0.007s
I20240503 14:12:19.696913 28600 raft_consensus.cc:348] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.697153 28600 raft_consensus.cc:374] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.697213 28600 raft_consensus.cc:729] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.697418 28600 consensus_queue.cc:260] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.697921 28600 ts_tablet_manager.cc:1398] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:19.698194 28600 tablet_bootstrap.cc:492] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:19.699680 28600 tablet_bootstrap.cc:654] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:19.701324 27856 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 63471.7us
I20240503 14:12:19.703819 28264 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
W20240503 14:12:19.686731 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2404807f8500437ababca53f6388b636 because the tablet was not found. No further retry: Not found: Tablet not found: 2404807f8500437ababca53f6388b636
I20240503 14:12:19.712013 29761 tablet.cc:1722] T c924a74ee836406fb3157baef6f4c901 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:19.713302 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7534f990ae504fa5aa364d6948a1760c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.714740 28653 ts_tablet_manager.cc:1909] T 60206e8e73e84e409ed6aba222d9ff4b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:19.715682 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 695) took 1828 ms. Trace:
W20240503 14:12:19.716493 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 218481d1ddd246cf9aa9c0e163666f4e consensus state has no leader
I20240503 14:12:19.717023 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3ba30dd208e14efca6ca2bcd7aab07c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.720027 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2cb9f4b3da44486fbb7fdbf277833634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:19.715786 28653 rpcz_store.cc:276] 0503 14:12:17.887254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.887409 (+ 155us) service_pool.cc:225] Handling call
0503 14:12:19.612563 (+1725154us) ts_tablet_manager.cc:1120] Deleting tablet 60206e8e73e84e409ed6aba222d9ff4b
0503 14:12:19.612577 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.643609 (+ 31032us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.674320 (+ 30711us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.715639 (+ 41319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1724595}
I20240503 14:12:19.721616 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a6709087fc714b21979fd6349c864785" } new_table_name: "default.Table_cd6c467bcf724c1f9e4a622d04d82bd1" modify_external_catalogs: true
I20240503 14:12:19.722026 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 697) took 1585 ms. Trace:
I20240503 14:12:19.722123 28653 rpcz_store.cc:276] 0503 14:12:18.136634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.137075 (+ 441us) service_pool.cc:225] Handling call
0503 14:12:19.721859 (+1584784us) ts_tablet_manager.cc:1120] Deleting tablet 215a6f53e47e49cb984565adf659614d
0503 14:12:19.721872 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.722003 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1584553}
I20240503 14:12:19.722550 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 698) took 1581 ms. Trace:
I20240503 14:12:19.718849 28028 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284] alter complete (version 1)
I20240503 14:12:19.722630 28653 rpcz_store.cc:276] 0503 14:12:18.140696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.140766 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:19.722381 (+1581615us) ts_tablet_manager.cc:1120] Deleting tablet 5cdc868ddf0a490292eae78ebc8c8ec6
0503 14:12:19.722392 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.722531 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1581354}
I20240503 14:12:19.722976 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 699) took 1582 ms. Trace:
I20240503 14:12:19.723053 28653 rpcz_store.cc:276] 0503 14:12:18.140843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.140902 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:19.722852 (+1581950us) ts_tablet_manager.cc:1120] Deleting tablet 833ad747feca411aa480dd420a6c02cc
0503 14:12:19.722860 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.722960 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1581672}
I20240503 14:12:19.723284 28653 tablet_replica.cc:330] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:19.723490 28653 raft_consensus.cc:2229] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.723701 28653 raft_consensus.cc:2258] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.725617 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet 2331aec871824a238f3f64de217d7092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:19.725914 28603 tablet_replica.cc:330] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:19.726133 28603 raft_consensus.cc:2229] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.726313 28603 raft_consensus.cc:2258] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.727351 28603 ts_tablet_manager.cc:1875] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.728319 29735 tablet.cc:1722] T c924a74ee836406fb3157baef6f4c901 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 14:12:19.732909 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ba30dd208e14efca6ca2bcd7aab07c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.733194 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cc84cf703048148abfe03b7dee12ce with delete_type TABLET_DATA_DELETED (replaced by tablet 5d308130f0dc4c16ac73058db332b41d) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.733587 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f89fca0480d4bf28160384ae0421bce with delete_type TABLET_DATA_DELETED (replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.733878 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 495190fca06a40a386c8215eb2195920 with delete_type TABLET_DATA_DELETED (replaced by tablet 7261088abbc641db8d737a48bbac25bd) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.734217 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1db04ab92c4af0847730a47635a697 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:19.734342 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7534f990ae504fa5aa364d6948a1760c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.734496 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 37364d5facc241f4bbccc979a50a0ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:19.734653 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cb9f4b3da44486fbb7fdbf277833634 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.734792 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 03700c1efc08471cb67dfe3a8016a740 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.735071 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.735389 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 9011de10a7ff497fb35ccb3ada7315ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:19.735832 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c5dfac02d4dfe932996d2a37fe031" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:19.736297 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dd4c914a4846bab3a00408247ec64a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:19.736497 28211 raft_consensus.cc:2454] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:19.745070 28600 tablet_bootstrap.cc:492] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:19.745249 28600 ts_tablet_manager.cc:1367] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.004s
I20240503 14:12:19.745935 28600 raft_consensus.cc:348] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.746151 28600 raft_consensus.cc:374] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.746208 28600 raft_consensus.cc:729] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.746455 28600 consensus_queue.cc:260] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.747114 28600 ts_tablet_manager.cc:1398] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:19.747423 28600 tablet_bootstrap.cc:492] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:19.749102 28600 tablet_bootstrap.cc:654] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.751943 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 2331aec871824a238f3f64de217d7092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:19.752175 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7abcb8658fc34c939aa74cf0bd981ae4 (table default.Table_80c5e40f9e494b2dbbbb7449a5b36569 [id=07297b58e7ca4fb1bf0e050a341d2c9c]) successfully deleted
I20240503 14:12:19.754468 28603 ts_tablet_manager.cc:1888] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.756018 28352 raft_consensus.cc:1261] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:19.757455 28603 log.cc:1196] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/2331aec871824a238f3f64de217d7092
I20240503 14:12:19.757841 29700 consensus_queue.cc:1035] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:19.758419 28603 ts_tablet_manager.cc:1909] T 2331aec871824a238f3f64de217d7092 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:19.752633 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e45df464f6954a56bec8b107b350320c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45df464f6954a56bec8b107b350320c already in progress: creating tablet
I20240503 14:12:19.762164 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 60206e8e73e84e409ed6aba222d9ff4b (table default.Table_e19b96f79a6c41c0bc266f1391941564 [id=c7144a78f2ae47599016aa64b6d0cbea]) successfully deleted
W20240503 14:12:19.762492 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 215a6f53e47e49cb984565adf659614d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215a6f53e47e49cb984565adf659614d already in progress: creating tablet
W20240503 14:12:19.762734 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cdc868ddf0a490292eae78ebc8c8ec6 already in progress: creating tablet
W20240503 14:12:19.763018 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 833ad747feca411aa480dd420a6c02cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 833ad747feca411aa480dd420a6c02cc already in progress: creating tablet
I20240503 14:12:19.763955 28653 ts_tablet_manager.cc:1875] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.766970 29761 tablet.cc:1722] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:19.771620 28600 tablet_bootstrap.cc:492] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:19.771809 28600 ts_tablet_manager.cc:1367] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.003s
I20240503 14:12:19.772562 28600 raft_consensus.cc:348] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.772794 28600 raft_consensus.cc:374] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.772862 28600 raft_consensus.cc:729] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.773955 28604 tablet_bootstrap.cc:492] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:19.774432 28604 ts_tablet_manager.cc:1367] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.213s user 0.003s sys 0.005s
I20240503 14:12:19.775584 28604 raft_consensus.cc:348] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.759699 29735 tablet.cc:1722] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:19.762029 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:19.789621 29735 tablet.cc:1722] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:19.764389 28614 tablet_bootstrap.cc:492] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:19.793165 28614 ts_tablet_manager.cc:1367] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.158s user 0.011s sys 0.009s
I20240503 14:12:19.795141 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 2331aec871824a238f3f64de217d7092 (table default.Table_9d9d27627b064475a2f86b5ddcf4236f [id=2d1971a7d562426886d85395dfa1b299]) successfully deleted
W20240503 14:12:19.796319 28427 consensus_peers.cc:485] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a32b6995818041778617a0d6bf936c8c. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.800582 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d59ca5509565404480a25ec78fe7194a consensus state has no leader
I20240503 14:12:19.800694 28210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:19.800992 28210 raft_consensus.cc:2454] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:19.797384 29788 tablet.cc:1722] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:19.801115 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9360be1891a744a48835f3976afd1059" } modify_external_catalogs: true
I20240503 14:12:19.776054 28604 raft_consensus.cc:374] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.801368 28604 raft_consensus.cc:729] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.801352 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9360be1891a744a48835f3976afd1059" } modify_external_catalogs: true
I20240503 14:12:19.801810 28604 consensus_queue.cc:260] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.802175 28427 leader_election.cc:302] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:19.802546 28604 ts_tablet_manager.cc:1398] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 14:12:19.802788 29701 raft_consensus.cc:2781] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:19.802917 29701 raft_consensus.cc:482] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:19.802964 28604 tablet_bootstrap.cc:492] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:19.803056 29701 raft_consensus.cc:3037] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.806144 29701 raft_consensus.cc:504] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.807022 29701 leader_election.cc:288] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:19.807746 29790 tablet.cc:1722] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:19.807960 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dd4c914a4846bab3a00408247ec64a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:19.808202 28352 raft_consensus.cc:3037] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.810787 28600 consensus_queue.cc:260] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.810832 28614 raft_consensus.cc:348] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.811700 28614 raft_consensus.cc:374] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:12:19.812289 28427 leader_election.cc:341] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.812507 28427 leader_election.cc:302] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:19.813047 29701 raft_consensus.cc:2726] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:19.815078 29786 tablet.cc:1722] T 8f0795253db548fb84c2d7bb3dc48335 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 14:12:19.816021 28427 consensus_peers.cc:485] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00b52ea336524beea2f255c8720f7472. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:19.817727 28600 ts_tablet_manager.cc:1398] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240503 14:12:19.818331 29701 consensus_queue.cc:1035] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:19.821452 28614 raft_consensus.cc:729] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.822686 28614 consensus_queue.cc:260] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.813015 28352 raft_consensus.cc:2454] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:19.825284 28427 leader_election.cc:302] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:19.825879 28614 ts_tablet_manager.cc:1398] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.032s user 0.002s sys 0.002s
I20240503 14:12:19.829012 28614 tablet_bootstrap.cc:492] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:19.829094 28661 ts_tablet_manager.cc:1888] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.829280 28661 log.cc:1196] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7218cefb13ea4983b2111ed5fd0c997b
I20240503 14:12:19.831923 28600 tablet_bootstrap.cc:492] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:19.838667 28614 tablet_bootstrap.cc:654] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:19.848199 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d04ce185de404299b4f66afd906ece80 consensus state has no leader
I20240503 14:12:19.848731 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_006d60d6900a406b8a5c73a795718f63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:19.849416 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_006d60d6900a406b8a5c73a795718f63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:19.850203 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d7b3acfa64194593899187c496e64c66" } modify_external_catalogs: true
I20240503 14:12:19.850471 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d7b3acfa64194593899187c496e64c66" } modify_external_catalogs: true
I20240503 14:12:19.851384 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f40d0e0c2ffb4c3a8169abdfdc3f8148" } modify_external_catalogs: true
I20240503 14:12:19.851410 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dd4c914a4846bab3a00408247ec64a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:19.851574 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f40d0e0c2ffb4c3a8169abdfdc3f8148" } modify_external_catalogs: true
I20240503 14:12:19.851624 28211 raft_consensus.cc:3037] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.854176 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d9f3904b328a47168ecb632d99de352b" } modify_external_catalogs: true
I20240503 14:12:19.854470 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d9f3904b328a47168ecb632d99de352b" } modify_external_catalogs: true
I20240503 14:12:19.854568 28211 raft_consensus.cc:2454] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:19.858435 28427 leader_election.cc:302] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:19.859398 29701 raft_consensus.cc:482] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.862511 29700 raft_consensus.cc:2781] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:19.862674 29700 raft_consensus.cc:482] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:19.862756 29700 raft_consensus.cc:3037] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.862838 28661 ts_tablet_manager.cc:1909] T 7218cefb13ea4983b2111ed5fd0c997b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:19.865535 29700 raft_consensus.cc:504] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.867107 29700 leader_election.cc:288] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:19.874416 28600 tablet_bootstrap.cc:654] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.878471 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:19.879171 28427 leader_election.cc:341] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.885897 28604 tablet_bootstrap.cc:654] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:19.888505 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 678) took 2772 ms. Trace:
I20240503 14:12:19.888741 28661 rpcz_store.cc:276] 0503 14:12:17.116634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.116734 (+ 100us) service_pool.cc:225] Handling call
0503 14:12:19.617962 (+2501228us) ts_tablet_manager.cc:1120] Deleting tablet 7218cefb13ea4983b2111ed5fd0c997b
0503 14:12:19.617973 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.769207 (+151234us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.828927 (+ 59720us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.888433 (+ 59506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2449214}
I20240503 14:12:19.889644 28661 tablet_replica.cc:330] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:19.889601 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7218cefb13ea4983b2111ed5fd0c997b (table default.Table_e3c825867cf84b72bfb708fcc39f6acb [id=3a24797cc53b439da44cc77025fa9f92]) successfully deleted
I20240503 14:12:19.890086 28661 raft_consensus.cc:2229] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:19.890350 28661 raft_consensus.cc:2258] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:19.891000 28661 ts_tablet_manager.cc:1875] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:19.901512 29750 raft_consensus.cc:2781] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:19.901983 29750 raft_consensus.cc:686] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:19.877055 29786 tablet.cc:1722] T c924a74ee836406fb3157baef6f4c901 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:19.885109 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:19.902590 29750 consensus_queue.cc:237] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.903131 28215 raft_consensus.cc:3037] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.905807 28604 tablet_bootstrap.cc:492] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:19.905969 28604 ts_tablet_manager.cc:1367] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20240503 14:12:19.906105 28215 raft_consensus.cc:2454] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:19.888650 29790 tablet.cc:1722] T 0e680f189add4b5c96de4046321aad2d P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:19.906798 28604 raft_consensus.cc:348] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.907078 28604 raft_consensus.cc:374] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.907145 28604 raft_consensus.cc:729] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.907363 28604 consensus_queue.cc:260] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.908694 29773 raft_consensus.cc:482] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.908869 29773 raft_consensus.cc:504] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.909279 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=402 overwritten=0 applied=402 ignored=0} inserts{seen=371 ignored=0} mutations{seen=442 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:19.909653 29773 leader_election.cc:288] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:19.910185 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: Bootstrap complete.
W20240503 14:12:19.910732 28290 leader_election.cc:341] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 140de4d395d54b7696062ed96b8aa89c
I20240503 14:12:19.907970 28427 leader_election.cc:302] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:19.910243 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140de4d395d54b7696062ed96b8aa89c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:19.916357 29794 raft_consensus.cc:2781] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:19.916391 28489 raft_consensus.cc:2454] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:19.916615 29794 raft_consensus.cc:686] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:19.917194 28290 leader_election.cc:302] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:19.917538 29773 raft_consensus.cc:2781] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:19.917646 29773 raft_consensus.cc:482] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:19.917709 29773 raft_consensus.cc:3037] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:19.921422 28600 tablet_bootstrap.cc:492] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:19.917258 29794 consensus_queue.cc:237] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.921593 28600 ts_tablet_manager.cc:1367] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20240503 14:12:19.922333 28600 raft_consensus.cc:348] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.922564 28600 raft_consensus.cc:374] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:19.922624 28600 raft_consensus.cc:729] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.923287 29773 raft_consensus.cc:504] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.923611 28653 ts_tablet_manager.cc:1888] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.924131 28653 log.cc:1196] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6c7c5dfac02d4dfe932996d2a37fe031
I20240503 14:12:19.924155 29773 leader_election.cc:288] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:19.925267 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140de4d395d54b7696062ed96b8aa89c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:19.925495 28489 raft_consensus.cc:3037] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:19.925724 28290 leader_election.cc:341] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 140de4d395d54b7696062ed96b8aa89c
I20240503 14:12:19.928761 28489 raft_consensus.cc:2454] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:19.929612 28290 leader_election.cc:302] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:19.930186 29773 raft_consensus.cc:2781] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:19.930454 29773 raft_consensus.cc:686] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:19.930768 29773 consensus_queue.cc:237] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.930523 29575 raft_consensus.cc:348] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:12:19.931854 29575 raft_consensus.cc:729] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10ddca38c26462eb221d6026e88c6cc, State: Initialized, Role: FOLLOWER
I20240503 14:12:19.934940 29575 consensus_queue.cc:260] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 402, Last appended: 1.402, Last appended by leader: 402, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:12:19.941005 29803 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:19.944494 29803 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:19.943476 29575 sys_catalog.cc:564] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:19.950049 28026 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_ea00e82e486041f49201fc0ef12fdd2d [id=8508dff5ad5b4cdf9f537c45c3ad1ed1] alter complete (version 1)
I20240503 14:12:19.952360 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f624029ec1b34039975e1aab597f5a1a" } modify_external_catalogs: true
I20240503 14:12:19.952629 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f624029ec1b34039975e1aab597f5a1a" } modify_external_catalogs: true
I20240503 14:12:19.956188 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 02dd4c914a4846bab3a00408247ec64a (table default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc]). Expected version 2 got 0
I20240503 14:12:19.956527 28653 ts_tablet_manager.cc:1909] T 6c7c5dfac02d4dfe932996d2a37fe031 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:19.943506 29786 tablet.cc:1722] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:19.947728 29790 tablet.cc:1722] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:19.953846 29772 raft_consensus.cc:482] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:19.958053 29772 raft_consensus.cc:504] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:19.958238 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8f0795253db548fb84c2d7bb3dc48335 (table default.Table_66a25fdf1a2147e8ba24818e5bdbabc4 [id=38e97234a2c74ca4bd6dafbb6cba5284]). Expected version 1 got 0
I20240503 14:12:19.958827 29772 leader_election.cc:288] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:19.960131 28327 tablet_service.cc:1467] Processing CreateTablet for tablet f9eadece4a0e4620abe5936f987f5404 (DEFAULT_TABLE table=default.Table_920c4ca573c64022b064774de3fea713 [id=2fe2fad7bb254d5cbfaff65f5284644e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.960638 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9eadece4a0e4620abe5936f987f5404. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.963263 28150 leader_election.cc:341] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:19.963672 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a92fc3833e49aaab8c5415de9bb685" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:19.964713 28029 catalog_manager.cc:5484] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:19.965307 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet e6c90869b5034128864faaf05697c230 (table default.Table_5dcdff74f1f7407084a45dc9fe588103 [id=47ade8624b00424d8fc04dbca5174ff4]). Expected version 3 got 0
I20240503 14:12:19.965927 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 708) took 1737 ms. Trace:
I20240503 14:12:19.969261 28469 tablet_service.cc:1467] Processing CreateTablet for tablet f9eadece4a0e4620abe5936f987f5404 (DEFAULT_TABLE table=default.Table_920c4ca573c64022b064774de3fea713 [id=2fe2fad7bb254d5cbfaff65f5284644e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.969697 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9eadece4a0e4620abe5936f987f5404. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:19.966171 28150 leader_election.cc:341] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:19.977198 28150 leader_election.cc:302] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:19.977778 29772 raft_consensus.cc:2726] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:19.978065 28653 rpcz_store.cc:276] 0503 14:12:18.228044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.228581 (+ 537us) service_pool.cc:225] Handling call
0503 14:12:19.723235 (+1494654us) ts_tablet_manager.cc:1120] Deleting tablet 6c7c5dfac02d4dfe932996d2a37fe031
0503 14:12:19.723245 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.767174 (+ 43929us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.923467 (+156293us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.965898 (+ 42431us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39589,"tablet-delete.queue_time_us":1494295}
I20240503 14:12:19.982208 28471 tablet_service.cc:1467] Processing CreateTablet for tablet af0ffe39a22045da94941d5c168373b7 (DEFAULT_TABLE table=default.Table_9dbad41900f744188c2984de6c2db121 [id=ebb2c2c33e9e4c248403d38fed26bb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.982471 28465 tablet_service.cc:1467] Processing CreateTablet for tablet f88e2264ce6b4c899d34ff889fee22e4 (DEFAULT_TABLE table=default.Table_3366802ae4a446d388f1679e5cc304fc [id=cbf0fc2016db4cb0896ba3aeb11b0f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.982758 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0ffe39a22045da94941d5c168373b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.983063 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 0b01014c6ce24f92be65bbf2ab410605 (DEFAULT_TABLE table=default.Table_83c9d941dac54f2b9d9b8798a7f56437 [id=8212f1aad18949ec81e871fa324f6f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.983410 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b01014c6ce24f92be65bbf2ab410605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.984382 28462 tablet_service.cc:1467] Processing CreateTablet for tablet efbcc25c83e8428bb25c5c52f2209072 (DEFAULT_TABLE table=default.Table_b2fcbb86236149c6865ae80f54e4cac3 [id=5c7d204e25384826b4a5715790f7f6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.984761 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbcc25c83e8428bb25c5c52f2209072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.985942 28461 tablet_service.cc:1467] Processing CreateTablet for tablet b86c2df0470a4945b3e8207f2d965de2 (DEFAULT_TABLE table=default.Table_f624029ec1b34039975e1aab597f5a1a [id=ec5ac613c0e64e61b942d1afabec59d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.986310 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86c2df0470a4945b3e8207f2d965de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.982208 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc8a83315b495d98d3a9f8de6a00c4 (DEFAULT_TABLE table=default.Table_a5e1a1d0ff44431fa10153a63904f373 [id=9ab320b8fb03485396e51eec30105f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.988390 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 9028a6990d0840ffa6849143f6c004db (DEFAULT_TABLE table=default.Table_717ce9be26654b1d9f243797798e0fa3 [id=d52865c20cea422f9cc1c5da654a7b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.988612 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc8a83315b495d98d3a9f8de6a00c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.988612 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 585fb08a8e744414b8abcf0e359c523d (DEFAULT_TABLE table=default.Table_bceb92591b5d4af8bf691e736575984d [id=7fc1013ce0034cf4b25458ca042e90d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.988888 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9028a6990d0840ffa6849143f6c004db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.988404 28473 tablet_service.cc:1467] Processing CreateTablet for tablet f6a8941aeb954bb3b1399e0276b0b79e (DEFAULT_TABLE table=default.Table_0a52cc1bc610449fba51167cb16d9be8 [id=6e4cde8fa23b411faee43a2661c3b407]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.989959 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a8941aeb954bb3b1399e0276b0b79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.990217 28661 ts_tablet_manager.cc:1888] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:19.990310 28661 log.cc:1196] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/47ba571a81eb4240a4a4a4a08cb879c5
I20240503 14:12:19.990607 28661 ts_tablet_manager.cc:1909] T 47ba571a81eb4240a4a4a4a08cb879c5 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:19.990929 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 679) took 2873 ms. Trace:
I20240503 14:12:19.991004 28661 rpcz_store.cc:276] 0503 14:12:17.156188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.156281 (+ 93us) service_pool.cc:225] Handling call
0503 14:12:19.889585 (+2733304us) ts_tablet_manager.cc:1120] Deleting tablet 47ba571a81eb4240a4a4a4a08cb879c5
0503 14:12:19.889596 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.894956 (+ 5360us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.990122 (+ 95166us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:19.990904 (+ 782us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2719656}
I20240503 14:12:19.991657 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 680) took 2835 ms. Trace:
I20240503 14:12:19.991729 28661 rpcz_store.cc:276] 0503 14:12:17.156420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.156475 (+ 55us) service_pool.cc:225] Handling call
0503 14:12:19.991498 (+2835023us) ts_tablet_manager.cc:1120] Deleting tablet ffcfc89a415e4c68b6a54318755fe0d7
0503 14:12:19.991508 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:19.991638 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2821196}
I20240503 14:12:19.991897 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88e2264ce6b4c899d34ff889fee22e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.992878 28330 tablet_service.cc:1467] Processing CreateTablet for tablet af0ffe39a22045da94941d5c168373b7 (DEFAULT_TABLE table=default.Table_9dbad41900f744188c2984de6c2db121 [id=ebb2c2c33e9e4c248403d38fed26bb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.992949 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 585fb08a8e744414b8abcf0e359c523d (DEFAULT_TABLE table=default.Table_bceb92591b5d4af8bf691e736575984d [id=7fc1013ce0034cf4b25458ca042e90d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.993238 28326 tablet_service.cc:1467] Processing CreateTablet for tablet f6a8941aeb954bb3b1399e0276b0b79e (DEFAULT_TABLE table=default.Table_0a52cc1bc610449fba51167cb16d9be8 [id=6e4cde8fa23b411faee43a2661c3b407]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.993367 28323 tablet_service.cc:1467] Processing CreateTablet for tablet efbcc25c83e8428bb25c5c52f2209072 (DEFAULT_TABLE table=default.Table_b2fcbb86236149c6865ae80f54e4cac3 [id=5c7d204e25384826b4a5715790f7f6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.993693 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc8a83315b495d98d3a9f8de6a00c4 (DEFAULT_TABLE table=default.Table_a5e1a1d0ff44431fa10153a63904f373 [id=9ab320b8fb03485396e51eec30105f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.994020 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 9028a6990d0840ffa6849143f6c004db (DEFAULT_TABLE table=default.Table_717ce9be26654b1d9f243797798e0fa3 [id=d52865c20cea422f9cc1c5da654a7b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.994364 28328 tablet_service.cc:1467] Processing CreateTablet for tablet f88e2264ce6b4c899d34ff889fee22e4 (DEFAULT_TABLE table=default.Table_3366802ae4a446d388f1679e5cc304fc [id=cbf0fc2016db4cb0896ba3aeb11b0f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.994711 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 0b01014c6ce24f92be65bbf2ab410605 (DEFAULT_TABLE table=default.Table_83c9d941dac54f2b9d9b8798a7f56437 [id=8212f1aad18949ec81e871fa324f6f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.992878 28336 tablet_service.cc:1467] Processing CreateTablet for tablet b86c2df0470a4945b3e8207f2d965de2 (DEFAULT_TABLE table=default.Table_f624029ec1b34039975e1aab597f5a1a [id=ec5ac613c0e64e61b942d1afabec59d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:19.995132 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585fb08a8e744414b8abcf0e359c523d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.995374 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0ffe39a22045da94941d5c168373b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.996464 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88e2264ce6b4c899d34ff889fee22e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:19.996381 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 47ba571a81eb4240a4a4a4a08cb879c5 (table default.Table_852b77a0ef2e4ee6bda7be721348aff8 [id=0a16a4e85e1842f1a2880d2873d363b1]) successfully deleted
W20240503 14:12:19.996831 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ffcfc89a415e4c68b6a54318755fe0d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffcfc89a415e4c68b6a54318755fe0d7 already in progress: creating tablet
I20240503 14:12:19.999713 28661 tablet_replica.cc:330] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.000559 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a8941aeb954bb3b1399e0276b0b79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.000536 28661 raft_consensus.cc:2229] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:20.000905 28661 raft_consensus.cc:2258] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.001482 28661 ts_tablet_manager.cc:1875] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.004042 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbcc25c83e8428bb25c5c52f2209072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.007850 28614 tablet_bootstrap.cc:492] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:20.008030 28614 ts_tablet_manager.cc:1367] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.181s user 0.011s sys 0.000s
I20240503 14:12:20.008564 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9028a6990d0840ffa6849143f6c004db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.009158 28614 raft_consensus.cc:348] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.009416 28614 raft_consensus.cc:374] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.009482 28614 raft_consensus.cc:729] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.010574 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 62efdb3c1b1243809fc37d078484c506 (DEFAULT_TABLE table=default.Table_1adea7aa88d4460dbc66959b5016f30b [id=d32639419b91402fb8202aec00beeca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.009761 28614 consensus_queue.cc:260] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.011005 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62efdb3c1b1243809fc37d078484c506. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.012789 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc8a83315b495d98d3a9f8de6a00c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.016997 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b01014c6ce24f92be65bbf2ab410605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.018218 28614 ts_tablet_manager.cc:1398] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:12:20.020431 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86c2df0470a4945b3e8207f2d965de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.020521 28604 ts_tablet_manager.cc:1398] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.114s user 0.005s sys 0.016s
I20240503 14:12:20.020860 28604 tablet_bootstrap.cc:492] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.022068 28322 tablet_service.cc:1467] Processing CreateTablet for tablet fce6d8cb8b514d7f920ccd2a4493c686 (DEFAULT_TABLE table=default.Table_6b74a6afc6cf44d19931b689292e8a36 [id=c4bbfc6e20cc48b19fd347bff79cf462]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.022508 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce6d8cb8b514d7f920ccd2a4493c686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.023618 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 62efdb3c1b1243809fc37d078484c506 (DEFAULT_TABLE table=default.Table_1adea7aa88d4460dbc66959b5016f30b [id=d32639419b91402fb8202aec00beeca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.023727 28614 tablet_bootstrap.cc:492] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:20.024520 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585fb08a8e744414b8abcf0e359c523d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.026783 29701 raft_consensus.cc:504] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:19.968794 28181 tablet_service.cc:1467] Processing CreateTablet for tablet f9eadece4a0e4620abe5936f987f5404 (DEFAULT_TABLE table=default.Table_920c4ca573c64022b064774de3fea713 [id=2fe2fad7bb254d5cbfaff65f5284644e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.028187 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9eadece4a0e4620abe5936f987f5404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.037129 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11137b75794e2c9dd61ded326c868f (DEFAULT_TABLE table=default.Table_6a788c3137364f1bbfef7c497254ecbf [id=6bb05c48537d42b9bdd3bf2fc7b3ad99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.037567 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e11137b75794e2c9dd61ded326c868f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.038187 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 29c78b82bd264b53ba60583e6d8f06d4 (DEFAULT_TABLE table=default.Table_b5f59b1cb2f440d4ae381befe0809756 [id=3cf0346f342943c89b600c0e8f1428e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.038653 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c78b82bd264b53ba60583e6d8f06d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.040874 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa32d7989ea4041b3323d620a14ca29 (DEFAULT_TABLE table=default.Table_087f6ee665a14f2a9f3cca94748dd55b [id=c208b65093af43a49231d3a3f3ea075a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.041085 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11137b75794e2c9dd61ded326c868f (DEFAULT_TABLE table=default.Table_6a788c3137364f1bbfef7c497254ecbf [id=6bb05c48537d42b9bdd3bf2fc7b3ad99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.041481 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e11137b75794e2c9dd61ded326c868f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.041574 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbf20a347f54724a0ce807436863785 (DEFAULT_TABLE table=default.Table_461c1f89931e46ed9efa3024088d2adb [id=5a6fdfe8c29f4ec18f4b46779fc1f499]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.041916 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbf20a347f54724a0ce807436863785. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.044247 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa32d7989ea4041b3323d620a14ca29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.045842 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa32d7989ea4041b3323d620a14ca29 (DEFAULT_TABLE table=default.Table_087f6ee665a14f2a9f3cca94748dd55b [id=c208b65093af43a49231d3a3f3ea075a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.046259 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa32d7989ea4041b3323d620a14ca29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.048754 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbf20a347f54724a0ce807436863785 (DEFAULT_TABLE table=default.Table_461c1f89931e46ed9efa3024088d2adb [id=5a6fdfe8c29f4ec18f4b46779fc1f499]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.049144 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbf20a347f54724a0ce807436863785. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.052328 29701 leader_election.cc:288] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.059657 28661 ts_tablet_manager.cc:1888] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:20.065757 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f0081cc2cb48aaa368741109032fd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:20.066916 28661 log.cc:1196] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a32b6995818041778617a0d6bf936c8c
I20240503 14:12:20.066825 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbf20a347f54724a0ce807436863785 (DEFAULT_TABLE table=default.Table_461c1f89931e46ed9efa3024088d2adb [id=5a6fdfe8c29f4ec18f4b46779fc1f499]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:20.066879 28427 leader_election.cc:341] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.066802 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa32d7989ea4041b3323d620a14ca29 (DEFAULT_TABLE table=default.Table_087f6ee665a14f2a9f3cca94748dd55b [id=c208b65093af43a49231d3a3f3ea075a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.066802 28604 tablet_bootstrap.cc:654] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.067458 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 29c78b82bd264b53ba60583e6d8f06d4 (DEFAULT_TABLE table=default.Table_b5f59b1cb2f440d4ae381befe0809756 [id=3cf0346f342943c89b600c0e8f1428e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.068408 28661 ts_tablet_manager.cc:1909] T a32b6995818041778617a0d6bf936c8c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.068326 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f0081cc2cb48aaa368741109032fd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:20.068572 28214 raft_consensus.cc:2454] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:20.069093 28427 leader_election.cc:302] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:20.069480 29701 raft_consensus.cc:2781] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:20.069634 29701 raft_consensus.cc:482] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:20.069787 29701 raft_consensus.cc:3037] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.070820 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8e6e922f5b4577861af5bdbee94f72 (DEFAULT_TABLE table=default.Table_cafee16021144efc9522addcc165ad89 [id=4bac486a404a43d1898b5af676b99df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.071249 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8e6e922f5b4577861af5bdbee94f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.073184 29701 raft_consensus.cc:504] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.074146 29701 leader_election.cc:288] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.069331 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 681) took 2864 ms. Trace:
I20240503 14:12:20.075929 28661 rpcz_store.cc:276] 0503 14:12:17.205168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.208203 (+ 3035us) service_pool.cc:225] Handling call
0503 14:12:19.999640 (+2791437us) ts_tablet_manager.cc:1120] Deleting tablet a32b6995818041778617a0d6bf936c8c
0503 14:12:19.999653 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.046309 (+ 46656us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.059527 (+ 13218us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.069307 (+ 9780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2779156}
I20240503 14:12:20.077351 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f0081cc2cb48aaa368741109032fd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:20.077534 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77f0081cc2cb48aaa368741109032fd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:20.077718 28214 raft_consensus.cc:3037] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.077912 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 689) took 2615 ms. Trace:
I20240503 14:12:20.078096 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbf20a347f54724a0ce807436863785. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.078032 28661 rpcz_store.cc:276] 0503 14:12:17.462106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.462159 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:20.077739 (+2615580us) ts_tablet_manager.cc:1120] Deleting tablet 16fed718124a4640967ab13a1ac2b379
0503 14:12:20.077755 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.077889 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2615336}
I20240503 14:12:20.080013 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 688) took 2618 ms. Trace:
I20240503 14:12:20.080953 28214 raft_consensus.cc:2454] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:20.081333 28661 rpcz_store.cc:276] 0503 14:12:17.461997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.462410 (+ 413us) service_pool.cc:225] Handling call
0503 14:12:20.079830 (+2617420us) ts_tablet_manager.cc:1120] Deleting tablet 007c84fd9d9247728b0babd48689aba8
0503 14:12:20.079846 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.079985 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2617240}
I20240503 14:12:20.085049 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8e6e922f5b4577861af5bdbee94f72 (DEFAULT_TABLE table=default.Table_cafee16021144efc9522addcc165ad89 [id=4bac486a404a43d1898b5af676b99df2]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:20.081504 28427 leader_election.cc:341] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.085467 28335 tablet_service.cc:1467] Processing CreateTablet for tablet f783ee62adba49579a0518b240b795bc (DEFAULT_TABLE table=default.Table_7efeff78f7b04eadbc81cf4b5fb025a1 [id=ad1eccda9b8b4c0d8d6da1544c583be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.085604 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8e6e922f5b4577861af5bdbee94f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.085886 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f783ee62adba49579a0518b240b795bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.086508 28328 tablet_service.cc:1467] Processing CreateTablet for tablet f3fa29325b6642eea48e2156202a7a0d (DEFAULT_TABLE table=default.Table_02def817ba4f4e739e340bed3ea01b0e [id=f8d47e798c5d41699688c920d7072bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.086583 28427 leader_election.cc:302] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:20.086874 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fa29325b6642eea48e2156202a7a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.086807 28473 tablet_service.cc:1467] Processing CreateTablet for tablet fce6d8cb8b514d7f920ccd2a4493c686 (DEFAULT_TABLE table=default.Table_6b74a6afc6cf44d19931b689292e8a36 [id=c4bbfc6e20cc48b19fd347bff79cf462]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.087059 29701 raft_consensus.cc:2781] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:20.087182 29701 raft_consensus.cc:686] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:20.087203 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce6d8cb8b514d7f920ccd2a4493c686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.088435 28334 tablet_service.cc:1467] Processing CreateTablet for tablet e814dbc2db654473b21aea5f04bca215 (DEFAULT_TABLE table=default.Table_4fba6a05d4a44929a210ee59b0d60ccf [id=638174bb9aeb4485b42e542404b22ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.088824 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e814dbc2db654473b21aea5f04bca215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.089372 28541 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:20.087426 29701 consensus_queue.cc:237] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.091526 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8ddcdb3c764dc59e9291725046d839 (DEFAULT_TABLE table=default.Table_40fb6c7d8a074d12962a5f9b17b3c14f [id=88def62c7cf54150bc60da5cb6ab78a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.091943 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8ddcdb3c764dc59e9291725046d839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.092020 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 690) took 2629 ms. Trace:
I20240503 14:12:20.092140 28661 rpcz_store.cc:276] 0503 14:12:17.462644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.462694 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:20.091860 (+2629166us) ts_tablet_manager.cc:1120] Deleting tablet 11d9f47543524fc5921c99aad869035f
0503 14:12:20.091872 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.091995 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2628973}
I20240503 14:12:20.092483 28186 tablet_service.cc:1467] Processing CreateTablet for tablet e814dbc2db654473b21aea5f04bca215 (DEFAULT_TABLE table=default.Table_4fba6a05d4a44929a210ee59b0d60ccf [id=638174bb9aeb4485b42e542404b22ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.092933 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e814dbc2db654473b21aea5f04bca215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.095130 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 691) took 2632 ms. Trace:
I20240503 14:12:20.095232 28661 rpcz_store.cc:276] 0503 14:12:17.462757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.462811 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:20.094962 (+2632151us) ts_tablet_manager.cc:1120] Deleting tablet e21a5dafa82f4660927d96c779a23478
0503 14:12:20.094976 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.095101 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2631983}
I20240503 14:12:20.097146 28458 tablet_service.cc:1467] Processing CreateTablet for tablet f3fa29325b6642eea48e2156202a7a0d (DEFAULT_TABLE table=default.Table_02def817ba4f4e739e340bed3ea01b0e [id=f8d47e798c5d41699688c920d7072bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.097684 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fa29325b6642eea48e2156202a7a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.097779 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 704) took 2227 ms. Trace:
I20240503 14:12:20.097705 28464 tablet_service.cc:1467] Processing CreateTablet for tablet e814dbc2db654473b21aea5f04bca215 (DEFAULT_TABLE table=default.Table_4fba6a05d4a44929a210ee59b0d60ccf [id=638174bb9aeb4485b42e542404b22ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.097872 28661 rpcz_store.cc:276] 0503 14:12:17.870298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.870392 (+ 94us) service_pool.cc:225] Handling call
0503 14:12:20.097666 (+2227274us) ts_tablet_manager.cc:1120] Deleting tablet 8c7d57fc4486457e9848daad79f04356
0503 14:12:20.097676 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.097760 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225966}
I20240503 14:12:20.098263 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e814dbc2db654473b21aea5f04bca215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.092483 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8ddcdb3c764dc59e9291725046d839 (DEFAULT_TABLE table=default.Table_40fb6c7d8a074d12962a5f9b17b3c14f [id=88def62c7cf54150bc60da5cb6ab78a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.099298 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8ddcdb3c764dc59e9291725046d839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.100140 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 706) took 2229 ms. Trace:
I20240503 14:12:20.100247 28661 rpcz_store.cc:276] 0503 14:12:17.870476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.870623 (+ 147us) service_pool.cc:225] Handling call
0503 14:12:20.099964 (+2229341us) ts_tablet_manager.cc:1120] Deleting tablet 5757e6e8703640519f7d38754cc0c8ce
0503 14:12:20.099976 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.100115 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228198}
I20240503 14:12:20.102691 28661 tablet_replica.cc:330] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.102922 28661 raft_consensus.cc:2229] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.103099 28661 raft_consensus.cc:2258] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.103125 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa32d7989ea4041b3323d620a14ca29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.103219 28462 tablet_service.cc:1467] Processing CreateTablet for tablet f783ee62adba49579a0518b240b795bc (DEFAULT_TABLE table=default.Table_7efeff78f7b04eadbc81cf4b5fb025a1 [id=ad1eccda9b8b4c0d8d6da1544c583be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.103331 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 380f267a984541c7878eeaa4eeee8f34 (DEFAULT_TABLE table=default.Table_5a4ea581b23745598fabca5d5e715f73 [id=7469579ac46f4390b9e6d43ffceaf7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.103595 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f783ee62adba49579a0518b240b795bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.103703 28661 ts_tablet_manager.cc:1875] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.103750 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f267a984541c7878eeaa4eeee8f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.107203 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 380f267a984541c7878eeaa4eeee8f34 (DEFAULT_TABLE table=default.Table_5a4ea581b23745598fabca5d5e715f73 [id=7469579ac46f4390b9e6d43ffceaf7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.107651 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f267a984541c7878eeaa4eeee8f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.108748 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11137b75794e2c9dd61ded326c868f (DEFAULT_TABLE table=default.Table_6a788c3137364f1bbfef7c497254ecbf [id=6bb05c48537d42b9bdd3bf2fc7b3ad99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.109222 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e11137b75794e2c9dd61ded326c868f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.111822 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8ddcdb3c764dc59e9291725046d839 (DEFAULT_TABLE table=default.Table_40fb6c7d8a074d12962a5f9b17b3c14f [id=88def62c7cf54150bc60da5cb6ab78a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.112329 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8ddcdb3c764dc59e9291725046d839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.112407 28183 tablet_service.cc:1467] Processing CreateTablet for tablet f783ee62adba49579a0518b240b795bc (DEFAULT_TABLE table=default.Table_7efeff78f7b04eadbc81cf4b5fb025a1 [id=ad1eccda9b8b4c0d8d6da1544c583be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.112833 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f783ee62adba49579a0518b240b795bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.113978 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 380f267a984541c7878eeaa4eeee8f34 (DEFAULT_TABLE table=default.Table_5a4ea581b23745598fabca5d5e715f73 [id=7469579ac46f4390b9e6d43ffceaf7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.114389 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f267a984541c7878eeaa4eeee8f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.116413 28178 tablet_service.cc:1467] Processing CreateTablet for tablet f3fa29325b6642eea48e2156202a7a0d (DEFAULT_TABLE table=default.Table_02def817ba4f4e739e340bed3ea01b0e [id=f8d47e798c5d41699688c920d7072bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.116873 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fa29325b6642eea48e2156202a7a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.116926 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c78b82bd264b53ba60583e6d8f06d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.121490 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8e6e922f5b4577861af5bdbee94f72 (DEFAULT_TABLE table=default.Table_cafee16021144efc9522addcc165ad89 [id=4bac486a404a43d1898b5af676b99df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.122030 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8e6e922f5b4577861af5bdbee94f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.106729 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 29c78b82bd264b53ba60583e6d8f06d4 (DEFAULT_TABLE table=default.Table_b5f59b1cb2f440d4ae381befe0809756 [id=3cf0346f342943c89b600c0e8f1428e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.124603 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c78b82bd264b53ba60583e6d8f06d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.129009 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62efdb3c1b1243809fc37d078484c506. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.130424 29750 raft_consensus.cc:482] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.130594 29750 raft_consensus.cc:504] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.131292 29750 leader_election.cc:288] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:19.966653 28600 consensus_queue.cc:260] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.001891 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 62efdb3c1b1243809fc37d078484c506 (DEFAULT_TABLE table=default.Table_1adea7aa88d4460dbc66959b5016f30b [id=d32639419b91402fb8202aec00beeca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.132225 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd8b2adaeeb4c3abfbfe68cc005e03a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:20.132553 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62efdb3c1b1243809fc37d078484c506. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:20.133427 28427 leader_election.cc:341] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.002244 28197 tablet_service.cc:1467] Processing CreateTablet for tablet b86c2df0470a4945b3e8207f2d965de2 (DEFAULT_TABLE table=default.Table_f624029ec1b34039975e1aab597f5a1a [id=ec5ac613c0e64e61b942d1afabec59d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.136432 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86c2df0470a4945b3e8207f2d965de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.002528 28182 tablet_service.cc:1467] Processing CreateTablet for tablet f6a8941aeb954bb3b1399e0276b0b79e (DEFAULT_TABLE table=default.Table_0a52cc1bc610449fba51167cb16d9be8 [id=6e4cde8fa23b411faee43a2661c3b407]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.002840 28185 tablet_service.cc:1467] Processing CreateTablet for tablet af0ffe39a22045da94941d5c168373b7 (DEFAULT_TABLE table=default.Table_9dbad41900f744188c2984de6c2db121 [id=ebb2c2c33e9e4c248403d38fed26bb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.003661 28187 tablet_service.cc:1467] Processing CreateTablet for tablet efbcc25c83e8428bb25c5c52f2209072 (DEFAULT_TABLE table=default.Table_b2fcbb86236149c6865ae80f54e4cac3 [id=5c7d204e25384826b4a5715790f7f6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.003662 28184 tablet_service.cc:1467] Processing CreateTablet for tablet f88e2264ce6b4c899d34ff889fee22e4 (DEFAULT_TABLE table=default.Table_3366802ae4a446d388f1679e5cc304fc [id=cbf0fc2016db4cb0896ba3aeb11b0f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.004024 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 585fb08a8e744414b8abcf0e359c523d (DEFAULT_TABLE table=default.Table_bceb92591b5d4af8bf691e736575984d [id=7fc1013ce0034cf4b25458ca042e90d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.004400 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc8a83315b495d98d3a9f8de6a00c4 (DEFAULT_TABLE table=default.Table_a5e1a1d0ff44431fa10153a63904f373 [id=9ab320b8fb03485396e51eec30105f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.004690 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 9028a6990d0840ffa6849143f6c004db (DEFAULT_TABLE table=default.Table_717ce9be26654b1d9f243797798e0fa3 [id=d52865c20cea422f9cc1c5da654a7b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.004993 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 0b01014c6ce24f92be65bbf2ab410605 (DEFAULT_TABLE table=default.Table_83c9d941dac54f2b9d9b8798a7f56437 [id=8212f1aad18949ec81e871fa324f6f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.022432 28194 tablet_service.cc:1467] Processing CreateTablet for tablet fce6d8cb8b514d7f920ccd2a4493c686 (DEFAULT_TABLE table=default.Table_6b74a6afc6cf44d19931b689292e8a36 [id=c4bbfc6e20cc48b19fd347bff79cf462]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.141542 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a8941aeb954bb3b1399e0276b0b79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.144304 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88e2264ce6b4c899d34ff889fee22e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.145447 28661 ts_tablet_manager.cc:1888] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.145612 28661 log.cc:1196] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e135b60660f448a7974520f2f87cf93a
I20240503 14:12:20.146039 28661 ts_tablet_manager.cc:1909] T e135b60660f448a7974520f2f87cf93a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.146423 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 702) took 2276 ms. Trace:
I20240503 14:12:20.146515 28661 rpcz_store.cc:276] 0503 14:12:17.870078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.874246 (+ 4168us) service_pool.cc:225] Handling call
0503 14:12:20.102628 (+2228382us) ts_tablet_manager.cc:1120] Deleting tablet e135b60660f448a7974520f2f87cf93a
0503 14:12:20.102640 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.113895 (+ 11255us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.145286 (+ 31391us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.146389 (+ 1103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228130}
I20240503 14:12:20.148265 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9028a6990d0840ffa6849143f6c004db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.151894 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b01014c6ce24f92be65bbf2ab410605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.152220 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbcc25c83e8428bb25c5c52f2209072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.141899 28614 tablet_bootstrap.cc:654] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.157317 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc8a83315b495d98d3a9f8de6a00c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.161073 29772 raft_consensus.cc:482] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.161298 29772 raft_consensus.cc:504] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.162025 29772 leader_election.cc:288] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:20.162631 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585fb08a8e744414b8abcf0e359c523d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.166437 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0ffe39a22045da94941d5c168373b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.168259 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce6d8cb8b514d7f920ccd2a4493c686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.172541 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:20.172816 28489 raft_consensus.cc:2454] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:20.176527 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:20.178757 28661 tablet_replica.cc:330] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.175989 28150 leader_election.cc:302] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:20.179306 28661 raft_consensus.cc:2229] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.179490 29811 raft_consensus.cc:2781] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:20.179620 28661 raft_consensus.cc:2258] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.179677 29811 raft_consensus.cc:482] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:20.179826 29811 raft_consensus.cc:3037] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.180454 28661 ts_tablet_manager.cc:1875] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:20.184010 28150 leader_election.cc:341] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.185707 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd8b2adaeeb4c3abfbfe68cc005e03a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:20.186060 28211 raft_consensus.cc:2454] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:20.184402 28604 tablet_bootstrap.cc:492] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.186355 29772 raft_consensus.cc:482] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.186465 29772 raft_consensus.cc:504] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.186700 28427 leader_election.cc:302] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:20.187274 29701 raft_consensus.cc:2781] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:20.187350 29772 leader_election.cc:288] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:20.187402 29701 raft_consensus.cc:482] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:20.187546 29701 raft_consensus.cc:3037] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.188104 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a081ba59ad44890bf2f94874c034277" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:20.188176 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a081ba59ad44890bf2f94874c034277" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:20.188817 28150 leader_election.cc:341] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:20.189043 28150 leader_election.cc:341] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.189124 28150 leader_election.cc:302] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:20.189656 29772 raft_consensus.cc:2726] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:20.190338 28600 ts_tablet_manager.cc:1398] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.269s user 0.002s sys 0.001s
I20240503 14:12:20.190652 28600 tablet_bootstrap.cc:492] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:20.191146 29701 raft_consensus.cc:504] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.192556 28604 ts_tablet_manager.cc:1367] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.172s user 0.009s sys 0.008s
I20240503 14:12:20.192603 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd8b2adaeeb4c3abfbfe68cc005e03a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:20.193311 28427 leader_election.cc:341] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.193490 28600 tablet_bootstrap.cc:654] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.193890 28604 raft_consensus.cc:348] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.194278 28604 raft_consensus.cc:374] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.194387 28604 raft_consensus.cc:729] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.194701 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 709) took 1966 ms. Trace:
I20240503 14:12:20.194820 28653 rpcz_store.cc:276] 0503 14:12:18.228222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.228996 (+ 774us) service_pool.cc:225] Handling call
0503 14:12:20.194552 (+1965556us) ts_tablet_manager.cc:1120] Deleting tablet 65cdd090bf0f493f8b92341369378532
0503 14:12:20.194564 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.194673 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1965336}
I20240503 14:12:20.195178 28653 tablet_replica.cc:330] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:20.195401 28653 raft_consensus.cc:2229] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.195571 28653 raft_consensus.cc:2258] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.195356 28604 consensus_queue.cc:260] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.196247 28653 ts_tablet_manager.cc:1875] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.196337 28604 ts_tablet_manager.cc:1398] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:20.084439 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a32b6995818041778617a0d6bf936c8c (table default.Table_ce1f05b05bd3429eacedb14a5f34b7da [id=e2f2ebf7731a41e3ae451eddf555c454]) successfully deleted
W20240503 14:12:20.200418 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 115949us
I20240503 14:12:20.200644 28604 tablet_bootstrap.cc:492] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:20.200770 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 16fed718124a4640967ab13a1ac2b379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16fed718124a4640967ab13a1ac2b379 already in progress: creating tablet
W20240503 14:12:20.201084 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 007c84fd9d9247728b0babd48689aba8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 007c84fd9d9247728b0babd48689aba8 already in progress: creating tablet
W20240503 14:12:20.201301 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 11d9f47543524fc5921c99aad869035f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11d9f47543524fc5921c99aad869035f already in progress: creating tablet
W20240503 14:12:20.201547 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e21a5dafa82f4660927d96c779a23478 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e21a5dafa82f4660927d96c779a23478 already in progress: creating tablet
W20240503 14:12:20.201747 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 8c7d57fc4486457e9848daad79f04356 because the tablet was not found. No further retry: Not found: Tablet not found: 8c7d57fc4486457e9848daad79f04356
I20240503 14:12:20.202317 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e135b60660f448a7974520f2f87cf93a (table default.Table_00e404138f154498b730db1de48ded05 [id=6edae5b6460546ae80b1f25ffec68d48]) successfully deleted
I20240503 14:12:20.202782 28604 tablet_bootstrap.cc:654] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:20.202804 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 888c28447db64751adca55ae769d0a37 (table default.Table_928c3ae55b104d519b4ad071f8babaac [id=bcaf1e8f01fc4906bf319ea69df40fab], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 888c28447db64751adca55ae769d0a37 consensus state has no leader
I20240503 14:12:20.203346 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6c7c5dfac02d4dfe932996d2a37fe031 (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
W20240503 14:12:20.203605 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 65cdd090bf0f493f8b92341369378532 because the tablet was not found. No further retry: Not found: Tablet not found: 65cdd090bf0f493f8b92341369378532
I20240503 14:12:20.221697 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd8b2adaeeb4c3abfbfe68cc005e03a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:20.222087 28211 raft_consensus.cc:3037] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.222481 29701 leader_election.cc:288] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.237445 29811 raft_consensus.cc:504] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.238226 29811 leader_election.cc:288] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:20.239181 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:20.239267 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:20.239395 28489 raft_consensus.cc:3037] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:20.239876 28150 leader_election.cc:341] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.241741 28211 raft_consensus.cc:2454] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:20.242337 28427 leader_election.cc:302] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:20.243580 29701 raft_consensus.cc:2781] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:20.244055 29701 raft_consensus.cc:686] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:20.244446 29701 consensus_queue.cc:237] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.242592 28489 raft_consensus.cc:2454] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:20.242892 28604 tablet_bootstrap.cc:492] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.252508 28604 ts_tablet_manager.cc:1367] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240503 14:12:20.252513 28150 leader_election.cc:302] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:20.253443 28604 raft_consensus.cc:348] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.253685 28604 raft_consensus.cc:374] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.253749 28604 raft_consensus.cc:729] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.254029 28604 consensus_queue.cc:260] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.254824 28604 ts_tablet_manager.cc:1398] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:20.242045 28661 ts_tablet_manager.cc:1888] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.256021 28661 log.cc:1196] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/b7d4e4a1f0b14406be957426a45bf45d
I20240503 14:12:20.257448 28661 ts_tablet_manager.cc:1909] T b7d4e4a1f0b14406be957426a45bf45d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.271235 29701 raft_consensus.cc:482] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.271454 29701 raft_consensus.cc:504] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.273520 29701 leader_election.cc:288] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.275003 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60206e8e73e84e409ed6aba222d9ff4b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:20.275822 28427 leader_election.cc:341] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:20.281188 28427 leader_election.cc:341] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 60206e8e73e84e409ed6aba222d9ff4b
I20240503 14:12:20.281363 28427 leader_election.cc:302] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:20.281863 29701 raft_consensus.cc:2726] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:20.296468 29811 raft_consensus.cc:2781] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:20.296646 29811 raft_consensus.cc:686] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:20.296898 29811 consensus_queue.cc:237] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.308190 28600 tablet_bootstrap.cc:492] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:20.308398 28600 ts_tablet_manager.cc:1367] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.001s
I20240503 14:12:20.309182 28600 raft_consensus.cc:348] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.309465 28600 raft_consensus.cc:374] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.309520 28600 raft_consensus.cc:729] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.309762 28600 consensus_queue.cc:260] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.310492 28600 ts_tablet_manager.cc:1398] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:20.310878 28600 tablet_bootstrap.cc:492] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:20.309839 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 105685us
I20240503 14:12:20.312700 28600 tablet_bootstrap.cc:654] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.322598 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9d3155335c5a46209c736c3c3ccd5eb4" } new_table_name: "default.Table_a9b093db5f6d4c709a15d7dc1ca6059b" modify_external_catalogs: true
I20240503 14:12:20.331064 28604 tablet_bootstrap.cc:492] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.331185 28653 ts_tablet_manager.cc:1888] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.331334 28653 log.cc:1196] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:20.332933 28653 ts_tablet_manager.cc:1909] T 0af0342e0e0b47499a9322b5b2cb3d03 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:20.333297 28604 tablet_bootstrap.cc:654] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.333859 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 710) took 2105 ms. Trace:
I20240503 14:12:20.333954 28653 rpcz_store.cc:276] 0503 14:12:18.228314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.229241 (+ 927us) service_pool.cc:225] Handling call
0503 14:12:20.195106 (+1965865us) ts_tablet_manager.cc:1120] Deleting tablet 0af0342e0e0b47499a9322b5b2cb3d03
0503 14:12:20.195117 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.199664 (+ 4547us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.331024 (+131360us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.333829 (+ 2805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1965640}
I20240503 14:12:20.334367 28653 tablet_replica.cc:330] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:20.334558 28653 raft_consensus.cc:2229] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.334720 28653 raft_consensus.cc:2258] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.335305 28653 ts_tablet_manager.cc:1875] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.337816 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 703) took 2467 ms. Trace:
I20240503 14:12:20.337971 28661 rpcz_store.cc:276] 0503 14:12:17.870211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.874525 (+ 4314us) service_pool.cc:225] Handling call
0503 14:12:20.178673 (+2304148us) ts_tablet_manager.cc:1120] Deleting tablet b7d4e4a1f0b14406be957426a45bf45d
0503 14:12:20.178689 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.236249 (+ 57560us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.241915 (+ 5666us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.337787 (+ 95872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2303961}
I20240503 14:12:20.340469 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 705) took 2470 ms. Trace:
I20240503 14:12:20.340591 28661 rpcz_store.cc:276] 0503 14:12:17.870391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:17.874732 (+ 4341us) service_pool.cc:225] Handling call
0503 14:12:20.338269 (+2463537us) ts_tablet_manager.cc:1120] Deleting tablet e45df464f6954a56bec8b107b350320c
0503 14:12:20.340282 (+ 2013us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.340443 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2463419}
I20240503 14:12:20.347123 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 708) took 2219 ms. Trace:
I20240503 14:12:20.347268 28661 rpcz_store.cc:276] 0503 14:12:18.127474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.128165 (+ 691us) service_pool.cc:225] Handling call
0503 14:12:20.340873 (+2212708us) ts_tablet_manager.cc:1120] Deleting tablet 215a6f53e47e49cb984565adf659614d
0503 14:12:20.346926 (+ 6053us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.347094 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2212440}
I20240503 14:12:20.348811 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 709) took 2212 ms. Trace:
I20240503 14:12:20.348940 28661 rpcz_store.cc:276] 0503 14:12:18.136817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.136908 (+ 91us) service_pool.cc:225] Handling call
0503 14:12:20.347571 (+2210663us) ts_tablet_manager.cc:1120] Deleting tablet 5cdc868ddf0a490292eae78ebc8c8ec6
0503 14:12:20.348628 (+ 1057us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.348780 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208690}
I20240503 14:12:20.349946 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 710) took 2212 ms. Trace:
I20240503 14:12:20.350085 28661 rpcz_store.cc:276] 0503 14:12:18.136987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.137114 (+ 127us) service_pool.cc:225] Handling call
0503 14:12:20.349228 (+2212114us) ts_tablet_manager.cc:1120] Deleting tablet 833ad747feca411aa480dd420a6c02cc
0503 14:12:20.349804 (+ 576us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.349923 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2210279}
I20240503 14:12:20.353314 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 714) took 2179 ms. Trace:
I20240503 14:12:20.353552 28661 rpcz_store.cc:276] 0503 14:12:18.173368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.196632 (+ 23264us) service_pool.cc:225] Handling call
0503 14:12:20.350372 (+2153740us) ts_tablet_manager.cc:1120] Deleting tablet 9aaf7d4927674eedbfd486df6a347f7c
0503 14:12:20.352887 (+ 2515us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.353283 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153319}
I20240503 14:12:20.354058 28661 tablet_replica.cc:330] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.354439 28661 raft_consensus.cc:2229] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.354893 28661 raft_consensus.cc:2258] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.357931 28600 tablet_bootstrap.cc:492] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:20.358141 28600 ts_tablet_manager.cc:1367] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240503 14:12:20.358597 28661 ts_tablet_manager.cc:1875] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.359081 28600 raft_consensus.cc:348] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.359331 28600 raft_consensus.cc:374] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.359386 28600 raft_consensus.cc:729] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.359609 28600 consensus_queue.cc:260] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.360136 28600 ts_tablet_manager.cc:1398] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:20.360503 28600 tablet_bootstrap.cc:492] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:20.362557 28600 tablet_bootstrap.cc:654] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.377075 28614 tablet_bootstrap.cc:492] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:20.377395 28614 ts_tablet_manager.cc:1367] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.354s user 0.025s sys 0.044s
I20240503 14:12:20.378196 28614 raft_consensus.cc:348] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.378536 28614 raft_consensus.cc:374] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.378638 28614 raft_consensus.cc:729] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.378942 28614 consensus_queue.cc:260] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.379768 28614 ts_tablet_manager.cc:1398] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:20.381175 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0af0342e0e0b47499a9322b5b2cb3d03 (table default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]) successfully deleted
I20240503 14:12:20.382190 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet b7d4e4a1f0b14406be957426a45bf45d (table default.Table_d3937b734026471d8da300845379e2dd [id=27afd28af9ee45a9ad0aad92898ce78c]) successfully deleted
W20240503 14:12:20.382705 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e45df464f6954a56bec8b107b350320c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45df464f6954a56bec8b107b350320c already in progress: creating tablet
W20240503 14:12:20.383065 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 215a6f53e47e49cb984565adf659614d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215a6f53e47e49cb984565adf659614d already in progress: creating tablet
W20240503 14:12:20.383359 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cdc868ddf0a490292eae78ebc8c8ec6 already in progress: creating tablet
I20240503 14:12:20.387130 28661 ts_tablet_manager.cc:1888] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.391112 28653 ts_tablet_manager.cc:1888] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.391405 28653 log.cc:1196] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/c5297664a4d943c28adfb6ac4f061680
I20240503 14:12:20.391825 28653 ts_tablet_manager.cc:1909] T c5297664a4d943c28adfb6ac4f061680 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:20.392308 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 711) took 2163 ms. Trace:
I20240503 14:12:20.397372 28653 rpcz_store.cc:276] 0503 14:12:18.228401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.229498 (+ 1097us) service_pool.cc:225] Handling call
0503 14:12:20.334308 (+2104810us) ts_tablet_manager.cc:1120] Deleting tablet c5297664a4d943c28adfb6ac4f061680
0503 14:12:20.334319 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.362752 (+ 28433us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.390973 (+ 28221us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.392283 (+ 1310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2104620}
I20240503 14:12:20.398313 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 703) took 2225 ms. Trace:
I20240503 14:12:20.398506 28653 rpcz_store.cc:276] 0503 14:12:18.173309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.173597 (+ 288us) service_pool.cc:225] Handling call
0503 14:12:20.398120 (+2224523us) ts_tablet_manager.cc:1120] Deleting tablet 9aaf7d4927674eedbfd486df6a347f7c
0503 14:12:20.398134 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.398290 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2137184}
I20240503 14:12:20.399195 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 713) took 2117 ms. Trace:
I20240503 14:12:20.399338 28653 rpcz_store.cc:276] 0503 14:12:18.281967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.284706 (+ 2739us) service_pool.cc:225] Handling call
0503 14:12:20.399041 (+2114335us) ts_tablet_manager.cc:1120] Deleting tablet cf810e84367f420b928a171790849445
0503 14:12:20.399053 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.399174 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2113557}
I20240503 14:12:20.399739 28614 tablet_bootstrap.cc:492] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:20.400068 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 716) took 2114 ms. Trace:
I20240503 14:12:20.400259 28653 rpcz_store.cc:276] 0503 14:12:18.286027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.286195 (+ 168us) service_pool.cc:225] Handling call
0503 14:12:20.399912 (+2113717us) ts_tablet_manager.cc:1120] Deleting tablet fc3f7c221cbc467dabfe5665ccd07d48
0503 14:12:20.399923 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.400050 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2113365}
I20240503 14:12:20.400760 28653 tablet_replica.cc:330] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:20.401024 28653 raft_consensus.cc:2229] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.401275 28653 raft_consensus.cc:2258] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.401752 28614 tablet_bootstrap.cc:654] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:20.403190 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 833ad747feca411aa480dd420a6c02cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 833ad747feca411aa480dd420a6c02cc already in progress: creating tablet
I20240503 14:12:20.403579 28653 ts_tablet_manager.cc:1875] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:20.403610 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9aaf7d4927674eedbfd486df6a347f7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aaf7d4927674eedbfd486df6a347f7c already in progress: creating tablet
I20240503 14:12:20.403954 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet c5297664a4d943c28adfb6ac4f061680 (table default.Table_78145a3b2c8543b5ab5a675ab4b08c91 [id=99ed67dc285f4dc69a9a9c6cec10b729]) successfully deleted
W20240503 14:12:20.404265 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9aaf7d4927674eedbfd486df6a347f7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aaf7d4927674eedbfd486df6a347f7c already in progress: creating tablet
W20240503 14:12:20.404559 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet cf810e84367f420b928a171790849445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf810e84367f420b928a171790849445 already in progress: creating tablet
W20240503 14:12:20.404832 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet fc3f7c221cbc467dabfe5665ccd07d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc3f7c221cbc467dabfe5665ccd07d48 already in progress: creating tablet
I20240503 14:12:20.405874 29773 raft_consensus.cc:482] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.406065 29773 raft_consensus.cc:504] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.406824 29773 leader_election.cc:288] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:20.399320 28604 tablet_bootstrap.cc:492] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.407330 28604 ts_tablet_manager.cc:1367] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.004s
I20240503 14:12:20.407660 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5e3649c71f4d4da710c69f85a82c25" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:20.407886 28489 raft_consensus.cc:2454] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:20.408035 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5e3649c71f4d4da710c69f85a82c25" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:20.408318 28211 raft_consensus.cc:2454] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:20.408219 28604 raft_consensus.cc:348] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.408440 28604 raft_consensus.cc:374] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.408495 28604 raft_consensus.cc:729] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.408749 28604 consensus_queue.cc:260] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.408960 28290 leader_election.cc:302] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:20.409459 28604 ts_tablet_manager.cc:1398] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:20.409584 28661 log.cc:1196] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/8b1728b6f0e64661aaa8184ce51366d2
I20240503 14:12:20.409837 29773 raft_consensus.cc:2781] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:20.409925 28661 ts_tablet_manager.cc:1909] T 8b1728b6f0e64661aaa8184ce51366d2 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.409933 29773 raft_consensus.cc:482] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:20.410001 29773 raft_consensus.cc:3037] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.411022 28604 tablet_bootstrap.cc:492] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.414666 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 719) took 2191 ms. Trace:
I20240503 14:12:20.414815 28661 rpcz_store.cc:276] 0503 14:12:18.222883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.222961 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:20.353998 (+2131037us) ts_tablet_manager.cc:1120] Deleting tablet 8b1728b6f0e64661aaa8184ce51366d2
0503 14:12:20.354009 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.358493 (+ 4484us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x61100058d1f0. stack: 00007f66f84cf756 00007f66f84cf510 00007f66f750248d 0000000000955804 00000000009557cd 000000000095519a 00007f670b970f9d 00007f6704eea05e 00007f6705017fe7 00007f6707aab57c 00007f6707add8d4 00007f6707abebc0 00007f6707abe94d 0000000000a60bcc 00007f66f8515b74 00007f66f851919b
0503 14:12:20.362498 (+ 4005us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.386987 (+ 24489us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.414620 (+ 27633us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6079232,"tablet-delete.queue_time_us":2130755}
I20240503 14:12:20.418001 28026 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_d73c3c2812414afc8bab352a368604cf [id=d1088a94cd0442c9bd9837bf9f0e1ff9] alter complete (version 3)
I20240503 14:12:20.419255 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet be238ee1d82548c9a1940e2a72e395c7
I20240503 14:12:20.422807 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 332ecc478e49424d9e7d7decbbc2c29c
I20240503 14:12:20.423873 28027 catalog_manager.cc:6554] ReplaceTablet: tablet 332ecc478e49424d9e7d7decbbc2c29c deleted and replaced by tablet d952b436198141ad80fef5fb1ea1e736
I20240503 14:12:20.416164 29773 raft_consensus.cc:504] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.431504 29773 leader_election.cc:288] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:20.432981 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5e3649c71f4d4da710c69f85a82c25" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:20.433302 28211 raft_consensus.cc:3037] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.433516 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5e3649c71f4d4da710c69f85a82c25" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:20.433728 28491 raft_consensus.cc:3037] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.435271 28661 tablet_replica.cc:330] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.435509 28661 raft_consensus.cc:2229] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:20.435882 28661 pending_rounds.cc:62] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:20.435938 28661 pending_rounds.cc:68] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597720881258496 op_type: NO_OP noop_request { }
I20240503 14:12:20.436098 28661 raft_consensus.cc:2866] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:20.436190 28661 raft_consensus.cc:2258] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.436630 28491 raft_consensus.cc:2454] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:20.436815 28661 ts_tablet_manager.cc:1875] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.436995 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet be238ee1d82548c9a1940e2a72e395c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.438982 28290 leader_election.cc:302] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:20.439584 29595 raft_consensus.cc:2781] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:20.439726 29595 raft_consensus.cc:686] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:20.440011 29595 consensus_queue.cc:237] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.441139 28614 tablet_bootstrap.cc:492] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:20.441300 28614 ts_tablet_manager.cc:1367] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240503 14:12:20.441918 29701 raft_consensus.cc:482] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.442137 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet be238ee1d82548c9a1940e2a72e395c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.442076 29701 raft_consensus.cc:504] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.442777 29701 leader_election.cc:288] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.443353 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 332ecc478e49424d9e7d7decbbc2c29c with delete_type TABLET_DATA_DELETED (replaced by tablet d952b436198141ad80fef5fb1ea1e736) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.444845 28614 raft_consensus.cc:348] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.444941 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 332ecc478e49424d9e7d7decbbc2c29c with delete_type TABLET_DATA_DELETED (replaced by tablet d952b436198141ad80fef5fb1ea1e736) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.445149 28614 raft_consensus.cc:374] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.445308 28614 raft_consensus.cc:729] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
W20240503 14:12:20.445451 28427 leader_election.cc:341] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 2331aec871824a238f3f64de217d7092
I20240503 14:12:20.445538 28614 consensus_queue.cc:260] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.441751 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 8b1728b6f0e64661aaa8184ce51366d2 (table default.Table_80d78826082349c882ea2c0ca66e0141 [id=bc70f87f0df24f569c3cc1c65a86f667]) successfully deleted
I20240503 14:12:20.457973 28604 tablet_bootstrap.cc:654] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.465133 28600 tablet_bootstrap.cc:492] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:20.465469 28600 ts_tablet_manager.cc:1367] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.006s
I20240503 14:12:20.467001 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet be238ee1d82548c9a1940e2a72e395c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.470567 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a31555fc800142bd849dbd572a194002
I20240503 14:12:20.471503 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet cf810e84367f420b928a171790849445 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:20.472445 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6615052ddbde487e840b6fd77a140e58
I20240503 14:12:20.472334 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5ba631c9bf2f4374991bda5d67c18af0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.473075 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ba631c9bf2f4374991bda5d67c18af0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.473264 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8796e63f1b454ff08f86549980fc079d" } new_table_name: "default.Table_522c1c58a4244b0b9b471bcf965a43f4" modify_external_catalogs: true
I20240503 14:12:20.474802 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f5bea5960cbb44d58d1dbabc888903c2
I20240503 14:12:20.475689 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet dcd405fd84d44076b1792b7fcbe00e84 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:20.466869 28600 raft_consensus.cc:348] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.476986 28600 raft_consensus.cc:374] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.477074 28600 raft_consensus.cc:729] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.477484 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b86c2df0470a4945b3e8207f2d965de2
I20240503 14:12:20.477625 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 332ecc478e49424d9e7d7decbbc2c29c with delete_type TABLET_DATA_DELETED (replaced by tablet d952b436198141ad80fef5fb1ea1e736) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.477864 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet a31555fc800142bd849dbd572a194002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.478587 28211 raft_consensus.cc:2454] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:20.478572 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c8a1b23185cf4c51ac94cb268e9e2222" } new_table_name: "default.Table_e5c58148b8e649f482d8025f0d340f3b" modify_external_catalogs: true
I20240503 14:12:20.479163 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 6615052ddbde487e840b6fd77a140e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.479754 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet b86c2df0470a4945b3e8207f2d965de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.485719 28604 tablet_bootstrap.cc:492] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.485949 28604 ts_tablet_manager.cc:1367] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.003s
W20240503 14:12:20.486552 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1132014802ed4c3682769ba86312eed2 consensus state has no leader
I20240503 14:12:20.487206 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1fafcfc2a1b747f0a7047b61caefb5df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.488025 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fafcfc2a1b747f0a7047b61caefb5df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.491336 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet e708be06de744743aac4e192bdd0f449 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:20.492467 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9aebe1270f6749608f4ec392cecede93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.493310 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9aebe1270f6749608f4ec392cecede93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.494108 28604 raft_consensus.cc:348] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.494369 28604 raft_consensus.cc:374] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.494422 28604 raft_consensus.cc:729] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.494624 28604 consensus_queue.cc:260] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.495215 28604 ts_tablet_manager.cc:1398] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.009s user 0.000s sys 0.000s
I20240503 14:12:20.495574 28604 tablet_bootstrap.cc:492] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:20.495987 28427 leader_election.cc:341] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2331aec871824a238f3f64de217d7092
I20240503 14:12:20.496165 28427 leader_election.cc:302] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:20.497249 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 6615052ddbde487e840b6fd77a140e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.497525 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bea5960cbb44d58d1dbabc888903c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.497913 28661 ts_tablet_manager.cc:1888] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:20.498034 28661 log.cc:1196] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/1603e2360041427f8635782e812b6431
I20240503 14:12:20.498378 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet b86c2df0470a4945b3e8207f2d965de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.498659 29750 raft_consensus.cc:482] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.498816 29750 raft_consensus.cc:504] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.499656 29750 leader_election.cc:288] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.499846 29750 raft_consensus.cc:2726] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:20.500391 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet a31555fc800142bd849dbd572a194002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.498386 28661 ts_tablet_manager.cc:1909] T 1603e2360041427f8635782e812b6431 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.500972 28614 ts_tablet_manager.cc:1398] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.060s user 0.008s sys 0.015s
I20240503 14:12:20.500957 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3ba30dd208e14efca6ca2bcd7aab07c8" } new_table_name: "default.Table_d6e534b5b7fc4dfcb5c7170d086552f8" modify_external_catalogs: true
I20240503 14:12:20.501549 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 720) took 2278 ms. Trace:
I20240503 14:12:20.501678 28661 rpcz_store.cc:276] 0503 14:12:18.223393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.223465 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:20.435207 (+2211742us) ts_tablet_manager.cc:1120] Deleting tablet 1603e2360041427f8635782e812b6431
0503 14:12:20.435220 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.477632 (+ 42412us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.497767 (+ 20135us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.501519 (+ 3752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2211530}
I20240503 14:12:20.502128 28661 tablet_replica.cc:330] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.502352 28661 raft_consensus.cc:2229] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.502596 28614 tablet_bootstrap.cc:492] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:20.486892 28600 consensus_queue.cc:260] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.502540 28661 raft_consensus.cc:2258] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.506182 28614 tablet_bootstrap.cc:654] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.506839 28661 ts_tablet_manager.cc:1875] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.507732 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_435f8c6696444d43a33900aee0682c9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.508630 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_435f8c6696444d43a33900aee0682c9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.502876 28604 tablet_bootstrap.cc:654] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.514560 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 1603e2360041427f8635782e812b6431 (table default.Table_cf24dd3520bd4373acfd719f2c7737ad [id=e0a3f887fca8477ba19c4ed3738b1955]) successfully deleted
I20240503 14:12:20.516146 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ced17a17c1af43d0be48f006c4b2a195" } new_table_name: "default.Table_10dee84bef6148bf8704fd1da1e39b58" modify_external_catalogs: true
W20240503 14:12:20.518105 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet be238ee1d82548c9a1940e2a72e395c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be238ee1d82548c9a1940e2a72e395c7 already in progress: creating tablet
W20240503 14:12:20.518522 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 332ecc478e49424d9e7d7decbbc2c29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 332ecc478e49424d9e7d7decbbc2c29c already in progress: creating tablet
I20240503 14:12:20.519578 28614 tablet_bootstrap.cc:492] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:20.520259 28614 ts_tablet_manager.cc:1367] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 14:12:20.520699 28336 tablet_service.cc:1514] Processing DeleteTablet for tablet a31555fc800142bd849dbd572a194002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.520730 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c97a43bd714da68c1a77408f3d1cd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:20.521761 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e708be06de744743aac4e192bdd0f449
I20240503 14:12:20.522418 28029 catalog_manager.cc:6554] ReplaceTablet: tablet e708be06de744743aac4e192bdd0f449 deleted and replaced by tablet d1ce025829c847baad0964cd131e4977
W20240503 14:12:20.523873 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 885dc7aabc044d10ac149432c8597212 consensus state has no leader
I20240503 14:12:20.523944 28614 raft_consensus.cc:348] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.524227 28614 raft_consensus.cc:374] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.524307 28614 raft_consensus.cc:729] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.524529 28614 consensus_queue.cc:260] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:20.525620 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e1f3443c298426db9ff5df1437884f9 has no consensus state
I20240503 14:12:20.526298 28614 ts_tablet_manager.cc:1398] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:20.526996 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet e708be06de744743aac4e192bdd0f449 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ce025829c847baad0964cd131e4977) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.527218 28614 tablet_bootstrap.cc:492] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:20.527844 28600 ts_tablet_manager.cc:1398] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20240503 14:12:20.528232 28600 tablet_bootstrap.cc:492] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:20.529495 28653 ts_tablet_manager.cc:1888] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.529633 28653 log.cc:1196] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/903f11a608b1483887d9b5234a404470
I20240503 14:12:20.529968 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bea5960cbb44d58d1dbabc888903c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.531847 28614 tablet_bootstrap.cc:654] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.532220 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet e708be06de744743aac4e192bdd0f449 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ce025829c847baad0964cd131e4977) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.533622 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c97a43bd714da68c1a77408f3d1cd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:20.534349 28214 raft_consensus.cc:2454] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:20.535823 28653 ts_tablet_manager.cc:1909] T 903f11a608b1483887d9b5234a404470 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:20.538902 28427 leader_election.cc:302] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:20.543637 29701 raft_consensus.cc:2781] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:20.543712 28604 tablet_bootstrap.cc:492] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.543776 29701 raft_consensus.cc:482] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:20.543851 29701 raft_consensus.cc:3037] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.543865 28604 ts_tablet_manager.cc:1367] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.000s
I20240503 14:12:20.544647 28604 raft_consensus.cc:348] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.544899 28604 raft_consensus.cc:374] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.544956 28604 raft_consensus.cc:729] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.545162 28604 consensus_queue.cc:260] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.545868 28604 ts_tablet_manager.cc:1398] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:20.546195 28604 tablet_bootstrap.cc:492] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.546775 29701 raft_consensus.cc:504] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.547564 29701 leader_election.cc:288] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.548195 28604 tablet_bootstrap.cc:654] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:20.550405 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10876f4512794e6aa72bda3c1da8b2d0 consensus state has no leader
I20240503 14:12:20.550904 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet cf221f17ddfd461f83f7f916d7833011 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:20.550845 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_178cd961b2bf465596a81dd8833ea327"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.551685 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_178cd961b2bf465596a81dd8833ea327 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.553834 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b676719e70fe4322adac2fa87c4cbd1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.554891 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b676719e70fe4322adac2fa87c4cbd1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.554113 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_521b7e40755f4748ad55fc13b690b642"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.557147 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_521b7e40755f4748ad55fc13b690b642 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.558743 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1ef8c263cd444fe989ea219bba7f5829"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.559621 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ef8c263cd444fe989ea219bba7f5829 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.560479 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f652e382ed2b40da8e5677659ce13416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.561357 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f652e382ed2b40da8e5677659ce13416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:20.562125 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c300b1f104949b3ae74f5b38d378a11 has no consensus state
I20240503 14:12:20.565568 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_34131ceab94648048a945b75d3042a55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.566478 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34131ceab94648048a945b75d3042a55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.563910 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet dcd405fd84d44076b1792b7fcbe00e84
I20240503 14:12:20.569365 28031 catalog_manager.cc:6554] ReplaceTablet: tablet dcd405fd84d44076b1792b7fcbe00e84 deleted and replaced by tablet 1eb6d931156b492e82140ee342cb1bbb
I20240503 14:12:20.574904 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6615052ddbde487e840b6fd77a140e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.576035 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 1eb6d931156b492e82140ee342cb1bbb) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.576272 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bea5960cbb44d58d1dbabc888903c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.580669 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet b86c2df0470a4945b3e8207f2d965de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.583416 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet e708be06de744743aac4e192bdd0f449 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ce025829c847baad0964cd131e4977) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.584787 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 1eb6d931156b492e82140ee342cb1bbb) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:20.589377 28427 leader_election.cc:341] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:20.590451 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a31555fc800142bd849dbd572a194002 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a31555fc800142bd849dbd572a194002 already in progress: creating tablet
I20240503 14:12:20.591260 28661 ts_tablet_manager.cc:1888] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.591410 28661 log.cc:1196] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/06fb79fcf30d485798c64c815e372dc1
W20240503 14:12:20.592259 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6615052ddbde487e840b6fd77a140e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6615052ddbde487e840b6fd77a140e58 already in progress: creating tablet
I20240503 14:12:20.598685 28604 tablet_bootstrap.cc:492] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.598861 28661 ts_tablet_manager.cc:1909] T 06fb79fcf30d485798c64c815e372dc1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.599005 28604 ts_tablet_manager.cc:1367] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.004s
I20240503 14:12:20.599740 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 721) took 2375 ms. Trace:
I20240503 14:12:20.599783 28604 raft_consensus.cc:348] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.600164 28604 raft_consensus.cc:374] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.600291 28604 raft_consensus.cc:729] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.600597 28604 consensus_queue.cc:260] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.600565 28024 catalog_manager.cc:5484] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:20.599864 28661 rpcz_store.cc:276] 0503 14:12:18.223781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.223832 (+ 51us) service_pool.cc:225] Handling call
0503 14:12:20.502065 (+2278233us) ts_tablet_manager.cc:1120] Deleting tablet 06fb79fcf30d485798c64c815e372dc1
0503 14:12:20.502078 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.566919 (+ 64841us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.591124 (+ 24205us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.599711 (+ 8587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2278073}
I20240503 14:12:20.601369 28604 ts_tablet_manager.cc:1398] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:20.601593 28661 tablet_replica.cc:330] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.601663 28604 tablet_bootstrap.cc:492] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.602466 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 06fb79fcf30d485798c64c815e372dc1 (table default.Table_3668987e06c74fa6bc566d6e46f92879 [id=92e89b0168d14b9a9fac6d7fca6be214]) successfully deleted
I20240503 14:12:20.603020 28472 tablet_service.cc:1467] Processing CreateTablet for tablet e44f92cd95164ed09ec0865bb87a6123 (DEFAULT_TABLE table=default.Table_7534f990ae504fa5aa364d6948a1760c [id=b23643b4d1804486ba3ead2cca24c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.603480 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44f92cd95164ed09ec0865bb87a6123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.601799 28661 raft_consensus.cc:2229] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.604391 28661 raft_consensus.cc:2258] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.604971 28661 ts_tablet_manager.cc:1875] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.608695 28604 tablet_bootstrap.cc:654] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.612303 29701 raft_consensus.cc:482] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.612505 29701 raft_consensus.cc:504] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.613250 29701 leader_election.cc:288] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.617200 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1f3443c298426db9ff5df1437884f9 (DEFAULT_TABLE table=default.Table_d6e534b5b7fc4dfcb5c7170d086552f8 [id=f9b141ce92884978b6e4689e37c54376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.617733 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1f3443c298426db9ff5df1437884f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.620815 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet cf221f17ddfd461f83f7f916d7833011
I20240503 14:12:20.620921 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 1199b9b0f7a44f5c8b841b09e300ce6e (DEFAULT_TABLE table=default.Table_deb8a08c99c54d0cbff9bcb72144aba6 [id=30030040cd5d4bb69ed506f27913bd07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.621402 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1199b9b0f7a44f5c8b841b09e300ce6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.621429 28025 catalog_manager.cc:6554] ReplaceTablet: tablet cf221f17ddfd461f83f7f916d7833011 deleted and replaced by tablet 5de9edfb0d064939bda2ec7a9e0e500b
I20240503 14:12:20.625205 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea9824e033540a1baac37ffb09b0aa8 (DEFAULT_TABLE table=default.Table_ae6805422d0d43b6acde2ee0ccb6a6d9 [id=0e12119f21b14e06b2458cf1bfd6e7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.625689 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea9824e033540a1baac37ffb09b0aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.627894 28604 tablet_bootstrap.cc:492] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.628047 28604 ts_tablet_manager.cc:1367] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20240503 14:12:20.628878 28604 raft_consensus.cc:348] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.629098 28604 raft_consensus.cc:374] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.629153 28604 raft_consensus.cc:729] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.629377 28604 consensus_queue.cc:260] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.629954 28604 ts_tablet_manager.cc:1398] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:20.630302 28604 tablet_bootstrap.cc:492] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.632306 28604 tablet_bootstrap.cc:654] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.636476 28661 ts_tablet_manager.cc:1888] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.636663 28661 log.cc:1196] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/2404807f8500437ababca53f6388b636
I20240503 14:12:20.637017 28661 ts_tablet_manager.cc:1909] T 2404807f8500437ababca53f6388b636 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.638026 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 722) took 2413 ms. Trace:
I20240503 14:12:20.638164 28661 rpcz_store.cc:276] 0503 14:12:18.224159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.224223 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:20.601544 (+2377321us) ts_tablet_manager.cc:1120] Deleting tablet 2404807f8500437ababca53f6388b636
0503 14:12:20.601552 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.616416 (+ 14864us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.636319 (+ 19903us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.637988 (+ 1669us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2377023}
I20240503 14:12:20.638564 28661 tablet_replica.cc:330] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.638800 28661 raft_consensus.cc:2229] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.638989 28661 raft_consensus.cc:2258] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.639639 28661 ts_tablet_manager.cc:1875] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.640838 28604 tablet_bootstrap.cc:492] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.641054 28604 ts_tablet_manager.cc:1367] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 14:12:20.641974 28604 raft_consensus.cc:348] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.642244 28604 raft_consensus.cc:374] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.642328 28604 raft_consensus.cc:729] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.642586 28604 consensus_queue.cc:260] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.643265 28604 ts_tablet_manager.cc:1398] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:20.643613 28604 tablet_bootstrap.cc:492] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.645856 29750 raft_consensus.cc:482] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.646028 29750 raft_consensus.cc:504] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.646682 29750 leader_election.cc:288] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.647444 28604 tablet_bootstrap.cc:654] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.656066 28604 tablet_bootstrap.cc:492] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.656396 28604 ts_tablet_manager.cc:1367] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240503 14:12:20.657449 28604 raft_consensus.cc:348] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.657790 28604 raft_consensus.cc:374] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.657903 28604 raft_consensus.cc:729] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.658210 28604 consensus_queue.cc:260] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.658947 28604 ts_tablet_manager.cc:1398] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:20.659364 28604 tablet_bootstrap.cc:492] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.660373 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6a788c3137364f1bbfef7c497254ecbf" } modify_external_catalogs: true
I20240503 14:12:20.660620 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6a788c3137364f1bbfef7c497254ecbf" } modify_external_catalogs: true
I20240503 14:12:20.661648 28604 tablet_bootstrap.cc:654] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.668551 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 4e11137b75794e2c9dd61ded326c868f
I20240503 14:12:20.669946 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 409cbf0cb8ef44819cfdb36f2fe83110 (DEFAULT_TABLE table=default.Table_e3b970f0246245259885e419a838a1a9 [id=fac1d13c6016463eba476f16c80ed7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.670562 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409cbf0cb8ef44819cfdb36f2fe83110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.671296 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9ccb39754c4d70a67d02d9cba8c3ef (DEFAULT_TABLE table=default.Table_06dc28f66ee6455e8cea52e834c0442c [id=1356071974124b03ba2fe1cacb66ed14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.671736 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9ccb39754c4d70a67d02d9cba8c3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.675050 28604 tablet_bootstrap.cc:492] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:20.675204 28604 ts_tablet_manager.cc:1367] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240503 14:12:20.676062 28604 raft_consensus.cc:348] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.676337 28604 raft_consensus.cc:374] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.676401 28604 raft_consensus.cc:729] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.676618 28604 consensus_queue.cc:260] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.677222 28604 ts_tablet_manager.cc:1398] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:20.678213 28661 ts_tablet_manager.cc:1888] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.679535 28661 log.cc:1196] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/65cdd090bf0f493f8b92341369378532
I20240503 14:12:20.679900 28661 ts_tablet_manager.cc:1909] T 65cdd090bf0f493f8b92341369378532 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.685413 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a65e77322444449cadf78698f2c98fd6" } modify_external_catalogs: true
I20240503 14:12:20.685789 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a65e77322444449cadf78698f2c98fd6" } modify_external_catalogs: true
I20240503 14:12:20.688588 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a52e2109f2f847dc992dd85cf261b6e3" } modify_external_catalogs: true
I20240503 14:12:20.689742 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a52e2109f2f847dc992dd85cf261b6e3" } modify_external_catalogs: true
I20240503 14:12:20.689455 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ddd5f026bb204de99edef3cc54bd8347" } modify_external_catalogs: true
I20240503 14:12:20.691210 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]). Expected version 3 got 0
I20240503 14:12:20.691746 28604 tablet_bootstrap.cc:492] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:20.694046 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 723) took 2469 ms. Trace:
I20240503 14:12:20.694327 28661 rpcz_store.cc:276] 0503 14:12:18.224546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.224626 (+ 80us) service_pool.cc:225] Handling call
0503 14:12:20.638500 (+2413874us) ts_tablet_manager.cc:1120] Deleting tablet 65cdd090bf0f493f8b92341369378532
0503 14:12:20.638514 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.655629 (+ 17115us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.678070 (+ 22441us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.694011 (+ 15941us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13150,"tablet-delete.queue_time_us":2413725}
I20240503 14:12:20.694865 28661 tablet_replica.cc:330] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.695462 28661 raft_consensus.cc:2229] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.696717 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_808ffc4c24d0480b87dab400bd4aec7e" } modify_external_catalogs: true
I20240503 14:12:20.697031 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_808ffc4c24d0480b87dab400bd4aec7e" } modify_external_catalogs: true
I20240503 14:12:20.697672 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a53279669bad40ad9243f4554a4e3c70
I20240503 14:12:20.699575 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_41af65ba4c3b41f5924502c36a902d9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:20.699261 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e5219472726e4702be79f785858e5ded" } new_table_name: "default.Table_53de0d5698bd46a3869b5d8c8a1a7233" modify_external_catalogs: true
I20240503 14:12:20.700384 28661 raft_consensus.cc:2258] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.701047 28661 ts_tablet_manager.cc:1875] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:20.701855 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41af65ba4c3b41f5924502c36a902d9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.703723 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ddd5f026bb204de99edef3cc54bd8347" } modify_external_catalogs: true
I20240503 14:12:20.705875 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6f0d676fa1e74ab89922f7ba607fd2d8
I20240503 14:12:20.706790 28028 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d1ce025829c847baad0964cd131e4977
W20240503 14:12:20.710577 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f5bea5960cbb44d58d1dbabc888903c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5bea5960cbb44d58d1dbabc888903c2 already in progress: creating tablet
W20240503 14:12:20.710988 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b86c2df0470a4945b3e8207f2d965de2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b86c2df0470a4945b3e8207f2d965de2 already in progress: creating tablet
W20240503 14:12:20.711421 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet dcd405fd84d44076b1792b7fcbe00e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcd405fd84d44076b1792b7fcbe00e84 already in progress: creating tablet
I20240503 14:12:20.700717 28024 catalog_manager.cc:5484] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:20.715739 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 02dd4c914a4846bab3a00408247ec64a (table default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc]). Expected version 2 got 0
I20240503 14:12:20.718991 28030 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_e9e796aab9294a76b536ba18bd97568c [id=d29fe68058a14e178e42a10043a91738] alter complete (version 3)
I20240503 14:12:20.719449 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a3d1314841574fc8bcf3ccb16755e421
I20240503 14:12:20.719455 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 353d55280fac4953a54347c3756adcee (DEFAULT_TABLE table=default.Table_f760a9cbcc20443ba5c28ec36efb98db [id=0dab2eec98884cadb02752b07c98e5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.719934 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353d55280fac4953a54347c3756adcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.720387 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf8f068f8424317bbfce6ff10bba61f (DEFAULT_TABLE table=default.Table_61e0a4e7d7544862b3028e7b3eec5bed [id=770381506d7c4730849f49d87eb5e05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.720783 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf8f068f8424317bbfce6ff10bba61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.721977 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 5d308130f0dc4c16ac73058db332b41d (DEFAULT_TABLE table=default.Table_20718ac286c9405b871c43b98ac94f8d [id=51930b6b93bd406581a479188de3423d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.722849 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c97a43bd714da68c1a77408f3d1cd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:20.725090 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ea32369d7dcc4ed5a82c2910a55b4583"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:20.727077 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b2fcbb86236149c6865ae80f54e4cac3" } new_table_name: "default.Table_c08f0e716f804f65a32618464e6bd962" modify_external_catalogs: true
I20240503 14:12:20.728479 28458 tablet_service.cc:1467] Processing CreateTablet for tablet 91eb4433685b4119a170054a0a777e05 (DEFAULT_TABLE table=default.Table_6bfb8beb36ba4f4292703a3b2224d553 [id=d056f8a1109d41edb1a921269c1134c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.731447 28458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91eb4433685b4119a170054a0a777e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.718820 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0362ba1c8df54ec6a99386f1777d063d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.732024 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea32369d7dcc4ed5a82c2910a55b4583 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:20.732134 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0362ba1c8df54ec6a99386f1777d063d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.732654 28471 tablet_service.cc:1467] Processing CreateTablet for tablet e04c0b6955b34267820ceecfffcfb88a (DEFAULT_TABLE table=default.Table_cc3d602d565c4c32b9127717dc19a658 [id=d9b6d299e57e4fc3a0e10f8304ef1ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.733090 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04c0b6955b34267820ceecfffcfb88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.733338 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d308130f0dc4c16ac73058db332b41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.734781 28329 tablet_service.cc:1467] Processing CreateTablet for tablet e44f92cd95164ed09ec0865bb87a6123 (DEFAULT_TABLE table=default.Table_7534f990ae504fa5aa364d6948a1760c [id=b23643b4d1804486ba3ead2cca24c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.735539 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc9815b8d349f2ac4b6d380d369638 (DEFAULT_TABLE table=default.Table_c0420d9a80af4738972115df9798052c [id=51f8df4cf4ae4790b97820f8f8f845d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.735759 29811 raft_consensus.cc:482] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.735934 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc9815b8d349f2ac4b6d380d369638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.737725 28465 tablet_service.cc:1467] Processing CreateTablet for tablet c92b0eec16a64dcdb0a5bee2a030e507 (DEFAULT_TABLE table=default.Table_e1abe3457b4248fb826e4c21037ceb83 [id=eedbf1330b194490a0d7c97e44b4216d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.738030 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet cf221f17ddfd461f83f7f916d7833011 with delete_type TABLET_DATA_DELETED (replaced by tablet 5de9edfb0d064939bda2ec7a9e0e500b) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.738101 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92b0eec16a64dcdb0a5bee2a030e507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.738355 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 2a691ceaa2b3464d83b722027ced0a96 (DEFAULT_TABLE table=default.Table_50e8eea84f564a4da088dd0efa136f69 [id=b996d4c5211c43b28d527f6b7501e99e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.738458 29772 raft_consensus.cc:482] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.738682 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a691ceaa2b3464d83b722027ced0a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.739327 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44f92cd95164ed09ec0865bb87a6123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.745004 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1f3443c298426db9ff5df1437884f9 (DEFAULT_TABLE table=default.Table_d6e534b5b7fc4dfcb5c7170d086552f8 [id=f9b141ce92884978b6e4689e37c54376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.745571 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1f3443c298426db9ff5df1437884f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.750627 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2404de92682c45eda574d316d0440be6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.751466 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2404de92682c45eda574d316d0440be6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.753939 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_34299e0857054fa5bc1b9b671c0842ba" } new_table_name: "default.Table_4afe0b312f964d3885779a752ec8c256" modify_external_catalogs: true
I20240503 14:12:20.754281 28327 tablet_service.cc:1467] Processing CreateTablet for tablet e04c0b6955b34267820ceecfffcfb88a (DEFAULT_TABLE table=default.Table_cc3d602d565c4c32b9127717dc19a658 [id=d9b6d299e57e4fc3a0e10f8304ef1ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.754472 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 409cbf0cb8ef44819cfdb36f2fe83110 (DEFAULT_TABLE table=default.Table_e3b970f0246245259885e419a838a1a9 [id=fac1d13c6016463eba476f16c80ed7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.754791 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04c0b6955b34267820ceecfffcfb88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.754854 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 91eb4433685b4119a170054a0a777e05 (DEFAULT_TABLE table=default.Table_6bfb8beb36ba4f4292703a3b2224d553 [id=d056f8a1109d41edb1a921269c1134c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.755060 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9ccb39754c4d70a67d02d9cba8c3ef (DEFAULT_TABLE table=default.Table_06dc28f66ee6455e8cea52e834c0442c [id=1356071974124b03ba2fe1cacb66ed14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.755198 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91eb4433685b4119a170054a0a777e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.755347 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9ccb39754c4d70a67d02d9cba8c3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.758291 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 353d55280fac4953a54347c3756adcee (DEFAULT_TABLE table=default.Table_f760a9cbcc20443ba5c28ec36efb98db [id=0dab2eec98884cadb02752b07c98e5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.758700 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353d55280fac4953a54347c3756adcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.754281 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea9824e033540a1baac37ffb09b0aa8 (DEFAULT_TABLE table=default.Table_ae6805422d0d43b6acde2ee0ccb6a6d9 [id=0e12119f21b14e06b2458cf1bfd6e7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.759471 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea9824e033540a1baac37ffb09b0aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.759699 29701 raft_consensus.cc:482] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.759847 29701 raft_consensus.cc:504] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.760524 29701 leader_election.cc:288] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.761936 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf8f068f8424317bbfce6ff10bba61f (DEFAULT_TABLE table=default.Table_61e0a4e7d7544862b3028e7b3eec5bed [id=770381506d7c4730849f49d87eb5e05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.762295 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf8f068f8424317bbfce6ff10bba61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.762246 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 244718a7d40d4ad0ac7c3a41bdb325d3 (DEFAULT_TABLE table=default.Table_6aea0eb4f439481da1da149c63f49de8 [id=32e1a0d216794a1085930a2cf694f88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.762825 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 0c300b1f104949b3ae74f5b38d378a11 (DEFAULT_TABLE table=default.Table_10dee84bef6148bf8704fd1da1e39b58 [id=7091b198d9a940cb8132bb92f5084114]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.754281 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 2a691ceaa2b3464d83b722027ced0a96 (DEFAULT_TABLE table=default.Table_50e8eea84f564a4da088dd0efa136f69 [id=b996d4c5211c43b28d527f6b7501e99e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.767314 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a691ceaa2b3464d83b722027ced0a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.768404 28469 tablet_service.cc:1467] Processing CreateTablet for tablet a1f53b98b55349a8ab937f58662b679f (DEFAULT_TABLE table=default.Table_b530238524784f869cd09190d81b9726 [id=cb87eb1b4c8c4d2487dae28d11b81b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.768857 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 1199b9b0f7a44f5c8b841b09e300ce6e (DEFAULT_TABLE table=default.Table_deb8a08c99c54d0cbff9bcb72144aba6 [id=30030040cd5d4bb69ed506f27913bd07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.770150 28470 tablet_service.cc:1467] Processing CreateTablet for tablet af0cf232932d48f880f7aced9c38b714 (DEFAULT_TABLE table=default.Table_2aeaeda9485f4eb1953af7e447a714ce [id=592270a7c17247eabc9fbb9195bb1a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.770599 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0cf232932d48f880f7aced9c38b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.770821 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d1314841574fc8bcf3ccb16755e421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.771534 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409cbf0cb8ef44819cfdb36f2fe83110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.775507 28455 tablet_service.cc:1514] Processing DeleteTablet for tablet a53279669bad40ad9243f4554a4e3c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.775931 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244718a7d40d4ad0ac7c3a41bdb325d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.776172 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f53b98b55349a8ab937f58662b679f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.779337 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8c51c0baf44430894429f6eaea797f (DEFAULT_TABLE table=default.Table_2cb9f4b3da44486fbb7fdbf277833634 [id=1aa5e872eb3846998d2fd9c1a511470e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.779738 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_990e0eda7c604928a02942994c0e4803" } new_table_name: "default.Table_41c48bdfa525455ea062c1d933c14447" modify_external_catalogs: true
I20240503 14:12:20.783206 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8c51c0baf44430894429f6eaea797f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.783725 28463 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11137b75794e2c9dd61ded326c868f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:20.784629 28427 leader_election.cc:341] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:20.784832 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d039e7aa896743fe87bb9365dfe8a704"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.785524 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d039e7aa896743fe87bb9365dfe8a704 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.785919 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.786756 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c300b1f104949b3ae74f5b38d378a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.787326 29773 raft_consensus.cc:482] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.787483 29773 raft_consensus.cc:504] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.788220 29773 leader_election.cc:288] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.790380 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 2404807f8500437ababca53f6388b636 (table default.Table_7d8f2c48beb5472fb05f0cbd475f7182 [id=a396cc6875ba41d29e801308d3f274d0]) successfully deleted
I20240503 14:12:20.790848 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 65cdd090bf0f493f8b92341369378532 (table default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c [id=d16dae55faab45f4bb6337e042735ad0]) successfully deleted
I20240503 14:12:20.788059 28456 tablet_service.cc:1467] Processing CreateTablet for tablet a53279669bad40ad9243f4554a4e3c70 (DEFAULT_TABLE table=default.Table_a65e77322444449cadf78698f2c98fd6 [id=b4e3102125404739b79234c6c7d1c7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.793565 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53279669bad40ad9243f4554a4e3c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.796952 28604 tablet_bootstrap.cc:654] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.797570 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_42ee02ce6afe4ddb8480ec52a7d8993f" } new_table_name: "default.Table_d97ddf0257014cd7835a63e1730e6907" modify_external_catalogs: true
I20240503 14:12:20.800282 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 5d308130f0dc4c16ac73058db332b41d (DEFAULT_TABLE table=default.Table_20718ac286c9405b871c43b98ac94f8d [id=51930b6b93bd406581a479188de3423d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:20.801697 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efbcc25c83e8428bb25c5c52f2209072 consensus state has no leader
I20240503 14:12:20.802282 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c1936a618b9f488fbd38c45bbc95303c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:20.803623 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3838859d5d884075bc288faedfd9ce98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:20.804965 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3f8dd5c5498a4ffda480e84d0526e3f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:20.806181 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e7b2865e9604ce58280321c7ee2a88e consensus state has no leader
I20240503 14:12:20.806711 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d8070970f59847d091fc2692c79d2ae9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:20.808858 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7b5c8fe5421e45faa3a85b72495c0cae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:20.810703 28661 ts_tablet_manager.cc:1888] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:20.810842 28661 log.cc:1196] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/daf5d153ab5142c0902cc6e96629f412
I20240503 14:12:20.811152 28661 ts_tablet_manager.cc:1909] T daf5d153ab5142c0902cc6e96629f412 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:20.812042 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 724) took 2587 ms. Trace:
I20240503 14:12:20.812184 28661 rpcz_store.cc:276] 0503 14:12:18.224929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.224982 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:20.694807 (+2469825us) ts_tablet_manager.cc:1120] Deleting tablet daf5d153ab5142c0902cc6e96629f412
0503 14:12:20.694818 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.763086 (+ 68268us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.810595 (+ 47509us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.812013 (+ 1418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2469682}
I20240503 14:12:20.812549 28661 tablet_replica.cc:330] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:20.812752 28661 raft_consensus.cc:2229] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:20.812914 28661 raft_consensus.cc:2258] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:20.813510 28661 ts_tablet_manager.cc:1875] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:20.815778 28030 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_85dbdd19a2514c63bc5ee54794385f56 [id=c2e9f50f96514774acb83f844cdd8692] alter complete (version 2)
I20240503 14:12:20.816670 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet cf221f17ddfd461f83f7f916d7833011 with delete_type TABLET_DATA_DELETED (replaced by tablet 5de9edfb0d064939bda2ec7a9e0e500b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.733554 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c97a43bd714da68c1a77408f3d1cd1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:20.817745 28214 raft_consensus.cc:3037] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:20.818760 28150 consensus_peers.cc:485] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:20.820371 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 1199b9b0f7a44f5c8b841b09e300ce6e (DEFAULT_TABLE table=default.Table_deb8a08c99c54d0cbff9bcb72144aba6 [id=30030040cd5d4bb69ed506f27913bd07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.821027 29772 raft_consensus.cc:504] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.821280 29811 raft_consensus.cc:504] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.826483 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 1eb6d931156b492e82140ee342cb1bbb) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:20.829104 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1f3443c298426db9ff5df1437884f9 (DEFAULT_TABLE table=default.Table_d6e534b5b7fc4dfcb5c7170d086552f8 [id=f9b141ce92884978b6e4689e37c54376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.829849 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 0c300b1f104949b3ae74f5b38d378a11 (DEFAULT_TABLE table=default.Table_10dee84bef6148bf8704fd1da1e39b58 [id=7091b198d9a940cb8132bb92f5084114]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.830394 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11137b75794e2c9dd61ded326c868f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.830807 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet a53279669bad40ad9243f4554a4e3c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.831133 28614 tablet_bootstrap.cc:492] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:20.831207 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.831303 28614 ts_tablet_manager.cc:1367] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.304s user 0.021s sys 0.034s
I20240503 14:12:20.831866 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d1314841574fc8bcf3ccb16755e421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:20.832441 28323 tablet_service.cc:1467] Processing CreateTablet for tablet c92b0eec16a64dcdb0a5bee2a030e507 (DEFAULT_TABLE table=default.Table_e1abe3457b4248fb826e4c21037ceb83 [id=eedbf1330b194490a0d7c97e44b4216d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.832866 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 244718a7d40d4ad0ac7c3a41bdb325d3 (DEFAULT_TABLE table=default.Table_6aea0eb4f439481da1da149c63f49de8 [id=32e1a0d216794a1085930a2cf694f88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.833005 28184 tablet_service.cc:1467] Processing CreateTablet for tablet 409cbf0cb8ef44819cfdb36f2fe83110 (DEFAULT_TABLE table=default.Table_e3b970f0246245259885e419a838a1a9 [id=fac1d13c6016463eba476f16c80ed7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.833213 28326 tablet_service.cc:1467] Processing CreateTablet for tablet a53279669bad40ad9243f4554a4e3c70 (DEFAULT_TABLE table=default.Table_a65e77322444449cadf78698f2c98fd6 [id=b4e3102125404739b79234c6c7d1c7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.833400 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea9824e033540a1baac37ffb09b0aa8 (DEFAULT_TABLE table=default.Table_ae6805422d0d43b6acde2ee0ccb6a6d9 [id=0e12119f21b14e06b2458cf1bfd6e7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.833559 28330 tablet_service.cc:1467] Processing CreateTablet for tablet af0cf232932d48f880f7aced9c38b714 (DEFAULT_TABLE table=default.Table_2aeaeda9485f4eb1953af7e447a714ce [id=592270a7c17247eabc9fbb9195bb1a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.833925 28325 tablet_service.cc:1467] Processing CreateTablet for tablet a1f53b98b55349a8ab937f58662b679f (DEFAULT_TABLE table=default.Table_b530238524784f869cd09190d81b9726 [id=cb87eb1b4c8c4d2487dae28d11b81b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.834425 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8c51c0baf44430894429f6eaea797f (DEFAULT_TABLE table=default.Table_2cb9f4b3da44486fbb7fdbf277833634 [id=1aa5e872eb3846998d2fd9c1a511470e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.835417 28183 tablet_service.cc:1467] Processing CreateTablet for tablet e44f92cd95164ed09ec0865bb87a6123 (DEFAULT_TABLE table=default.Table_7534f990ae504fa5aa364d6948a1760c [id=b23643b4d1804486ba3ead2cca24c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.835825 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet daf5d153ab5142c0902cc6e96629f412 (table default.Table_6673b8d0a18d42c3a18f6e611c596580 [id=3c43a0fcf51143179a1ed0e1b8e333c1]) successfully deleted
I20240503 14:12:20.836294 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 718) took 2439 ms. Trace:
W20240503 14:12:20.839653 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f8dd5c5498a4ffda480e84d0526e3f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.841221 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9ccb39754c4d70a67d02d9cba8c3ef (DEFAULT_TABLE table=default.Table_06dc28f66ee6455e8cea52e834c0442c [id=1356071974124b03ba2fe1cacb66ed14]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:20.841323 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b5c8fe5421e45faa3a85b72495c0cae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.841729 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1199b9b0f7a44f5c8b841b09e300ce6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:20.842767 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1936a618b9f488fbd38c45bbc95303c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.842983 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9ccb39754c4d70a67d02d9cba8c3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.843006 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
W20240503 14:12:20.844166 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3838859d5d884075bc288faedfd9ce98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.844385 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44f92cd95164ed09ec0865bb87a6123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.844334 28189 tablet_service.cc:1467] Processing CreateTablet for tablet e04c0b6955b34267820ceecfffcfb88a (DEFAULT_TABLE table=default.Table_cc3d602d565c4c32b9127717dc19a658 [id=d9b6d299e57e4fc3a0e10f8304ef1ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.844738 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04c0b6955b34267820ceecfffcfb88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.844792 29811 leader_election.cc:288] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:20.844990 29811 consensus_queue.cc:1035] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:20.845407 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8070970f59847d091fc2692c79d2ae9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:20.847254 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc9815b8d349f2ac4b6d380d369638 (DEFAULT_TABLE table=default.Table_c0420d9a80af4738972115df9798052c [id=51f8df4cf4ae4790b97820f8f8f845d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.848359 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 353d55280fac4953a54347c3756adcee (DEFAULT_TABLE table=default.Table_f760a9cbcc20443ba5c28ec36efb98db [id=0dab2eec98884cadb02752b07c98e5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.848466 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495190fca06a40a386c8215eb2195920" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:20.846568 28614 raft_consensus.cc:348] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.849047 28614 raft_consensus.cc:374] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.849112 28614 raft_consensus.cc:729] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.850982 28614 consensus_queue.cc:260] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.854753 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abcb8658fc34c939aa74cf0bd981ae4" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:20.874919 28614 ts_tablet_manager.cc:1398] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.043s user 0.002s sys 0.002s
W20240503 14:12:20.877962 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86804987bdf340dc83ddd58ba2bc8a96 consensus state has no leader
I20240503 14:12:20.878938 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea9824e033540a1baac37ffb09b0aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.880581 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b9af7d02fc5a42a7a09ecbfb33775ccf" } modify_external_catalogs: true
I20240503 14:12:20.880863 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b9af7d02fc5a42a7a09ecbfb33775ccf" } modify_external_catalogs: true
I20240503 14:12:20.881058 29701 raft_consensus.cc:482] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.881227 29701 raft_consensus.cc:504] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.882956 28653 rpcz_store.cc:276] 0503 14:12:18.386605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.386853 (+ 248us) service_pool.cc:225] Handling call
0503 14:12:20.400704 (+2013851us) ts_tablet_manager.cc:1120] Deleting tablet 903f11a608b1483887d9b5234a404470
0503 14:12:20.400714 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:20.464530 (+ 63816us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.529369 (+ 64839us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:20.826293 (+296924us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2013548}
I20240503 14:12:20.883728 29772 leader_election.cc:288] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:20.882766 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 91eb4433685b4119a170054a0a777e05 (DEFAULT_TABLE table=default.Table_6bfb8beb36ba4f4292703a3b2224d553 [id=d056f8a1109d41edb1a921269c1134c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.883509 28614 tablet_bootstrap.cc:492] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:20.885246 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92b0eec16a64dcdb0a5bee2a030e507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.885289 28194 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf8f068f8424317bbfce6ff10bba61f (DEFAULT_TABLE table=default.Table_61e0a4e7d7544862b3028e7b3eec5bed [id=770381506d7c4730849f49d87eb5e05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.885537 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53279669bad40ad9243f4554a4e3c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.885958 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0cf232932d48f880f7aced9c38b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.888248 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8c51c0baf44430894429f6eaea797f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.888703 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244718a7d40d4ad0ac7c3a41bdb325d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.890458 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f53b98b55349a8ab937f58662b679f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.892150 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c300b1f104949b3ae74f5b38d378a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.896014 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1199b9b0f7a44f5c8b841b09e300ce6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:20.897866 28290 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 53236.7us
I20240503 14:12:20.899796 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc9815b8d349f2ac4b6d380d369638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.904585 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d308130f0dc4c16ac73058db332b41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.908983 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6de7dc45944845628d02aa20bfa0dcad" } modify_external_catalogs: true
I20240503 14:12:20.910634 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.914448 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 1603e2360041427f8635782e812b6431 with delete_type TABLET_DATA_DELETED (replaced by tablet 31ddd791d4e14630bfb3b4b1513f76e1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:20.930202 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d69cc969ea949a684a7e5aae237c13e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:20.930521 28491 raft_consensus.cc:2379] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:20.931828 29701 leader_election.cc:288] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.932435 28614 tablet_bootstrap.cc:654] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.943702 28614 tablet_bootstrap.cc:492] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:20.944406 28614 ts_tablet_manager.cc:1367] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.002s
I20240503 14:12:20.945791 28614 raft_consensus.cc:348] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.948614 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 8097ba5606cf42d68448833de36d85c7 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:20.946329 28290 leader_election.cc:302] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:20.948817 28614 raft_consensus.cc:374] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.948913 28614 raft_consensus.cc:729] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.949074 29773 raft_consensus.cc:3037] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.949998 28614 consensus_queue.cc:260] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.951792 29773 raft_consensus.cc:2726] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:20.952816 28614 ts_tablet_manager.cc:1398] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:12:20.953639 28614 tablet_bootstrap.cc:492] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:20.952874 29773 consensus_queue.cc:1035] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:20.957465 28614 tablet_bootstrap.cc:654] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:20.958719 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0a7f788e080745b2bf2fe782dfae1bbd" } modify_external_catalogs: true
I20240503 14:12:20.959831 29701 raft_consensus.cc:482] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:20.959978 29701 raft_consensus.cc:504] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:20.960606 29701 leader_election.cc:288] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:20.960827 28406 heartbeater.cc:347] Connected to a master server at 127.27.50.253:44259
I20240503 14:12:20.959206 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0a7f788e080745b2bf2fe782dfae1bbd" } modify_external_catalogs: true
W20240503 14:12:20.980413 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet cf221f17ddfd461f83f7f916d7833011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf221f17ddfd461f83f7f916d7833011 already in progress: creating tablet
W20240503 14:12:20.984361 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 4e11137b75794e2c9dd61ded326c868f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e11137b75794e2c9dd61ded326c868f already in progress: creating tablet
I20240503 14:12:20.987896 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:20.988312 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a53279669bad40ad9243f4554a4e3c70 because the tablet was not found. No further retry: Not found: Tablet not found: a53279669bad40ad9243f4554a4e3c70
I20240503 14:12:20.993017 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f89fca0480d4bf28160384ae0421bce with delete_type TABLET_DATA_DELETED (replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:20.993861 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f0d676fa1e74ab89922f7ba607fd2d8 already in progress: creating tablet
I20240503 14:12:20.995379 28614 tablet_bootstrap.cc:492] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:20.995599 28614 ts_tablet_manager.cc:1367] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.008s
I20240503 14:12:20.996484 28614 raft_consensus.cc:348] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.996860 28614 raft_consensus.cc:374] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:20.996966 28614 raft_consensus.cc:729] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
W20240503 14:12:20.996358 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a3d1314841574fc8bcf3ccb16755e421 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3d1314841574fc8bcf3ccb16755e421 already in progress: creating tablet
I20240503 14:12:20.997530 28614 consensus_queue.cc:260] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:20.998493 28614 ts_tablet_manager.cc:1398] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:20.999532 28614 tablet_bootstrap.cc:492] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.002501 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903f11a608b1483887d9b5234a404470" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.002725 28355 raft_consensus.cc:2454] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:21.002931 29773 raft_consensus.cc:482] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.003086 29773 raft_consensus.cc:504] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.003788 29773 leader_election.cc:288] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.003909 28614 tablet_bootstrap.cc:654] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.005216 29773 raft_consensus.cc:482] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.005362 29773 raft_consensus.cc:504] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.006076 29773 leader_election.cc:288] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:21.006960 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903f11a608b1483887d9b5234a404470" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.007230 28493 raft_consensus.cc:2454] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:21.009487 28427 consensus_peers.cc:485] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:21.009939 28427 leader_election.cc:341] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 2f89fca0480d4bf28160384ae0421bce
W20240503 14:12:21.010370 28427 leader_election.cc:341] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:21.011358 28427 leader_election.cc:341] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:21.012015 28427 consensus_peers.cc:485] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:21.012468 28427 leader_election.cc:341] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:21.012859 28427 leader_election.cc:302] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:21.013386 29845 raft_consensus.cc:2781] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.013465 29845 raft_consensus.cc:482] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.013512 28614 tablet_bootstrap.cc:492] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.013525 29845 raft_consensus.cc:3037] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.013644 28614 ts_tablet_manager.cc:1367] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20240503 14:12:21.014375 28614 raft_consensus.cc:348] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.014586 28614 raft_consensus.cc:374] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.014631 28614 raft_consensus.cc:729] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.014855 28614 consensus_queue.cc:260] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.015434 28614 ts_tablet_manager.cc:1398] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:21.015705 28614 tablet_bootstrap.cc:492] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.016337 29845 raft_consensus.cc:504] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.017108 29845 leader_election.cc:288] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.017591 28614 tablet_bootstrap.cc:654] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.021971 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d9985eecda4f3194c3251216aa58ee" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.023154 28290 leader_election.cc:302] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:21.025612 29773 raft_consensus.cc:2781] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:12:21.025653 28290 leader_election.cc:341] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:21.025730 29773 raft_consensus.cc:482] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.025807 29773 raft_consensus.cc:3037] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:20.909564 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6de7dc45944845628d02aa20bfa0dcad" } modify_external_catalogs: true
I20240503 14:12:21.031862 29773 raft_consensus.cc:504] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.033661 28604 tablet_bootstrap.cc:492] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:21.034039 28604 ts_tablet_manager.cc:1367] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.342s user 0.003s sys 0.016s
I20240503 14:12:21.039523 29773 leader_election.cc:288] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.039820 29845 raft_consensus.cc:482] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.040066 29845 raft_consensus.cc:504] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.040992 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903f11a608b1483887d9b5234a404470" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:21.041244 28493 raft_consensus.cc:2379] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:21.041968 29845 leader_election.cc:288] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.044462 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0d81fa49986c4ca99528a823b4a22ae6" } modify_external_catalogs: true
I20240503 14:12:21.044824 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0d81fa49986c4ca99528a823b4a22ae6" } modify_external_catalogs: true
I20240503 14:12:21.046532 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b80186a1b50741aca2ee3ff55a4ef6be" } new_table_name: "default.Table_afdf531aa97a45ab9f8c7757d67b3523" modify_external_catalogs: true
I20240503 14:12:21.049440 28024 catalog_manager.cc:5484] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:21.056778 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d69cc969ea949a684a7e5aae237c13e consensus state has no leader
I20240503 14:12:21.057363 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_55224d5639254fa1a4890def9934e40a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.057991 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55224d5639254fa1a4890def9934e40a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.060457 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6c998e518541248a87e4c1551ec080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.060634 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.060901 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_eeca27b59dd543faa1ea8c60944ddadf" } modify_external_catalogs: true
I20240503 14:12:21.060972 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.061108 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_eeca27b59dd543faa1ea8c60944ddadf" } modify_external_catalogs: true
I20240503 14:12:21.061590 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet c5297664a4d943c28adfb6ac4f061680 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.069474 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410e8b6b511403ea609b1353855937b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.070434 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet bf21f6234dd64cf8a32f1599fe50746e with delete_type TABLET_DATA_DELETED (replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.078224 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 3366f813bd1f4df3843487ad40ba4203 (DEFAULT_TABLE table=default.Table_f652e382ed2b40da8e5677659ce13416 [id=c07d28d716c14843b03e915b3a3023c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.078732 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3366f813bd1f4df3843487ad40ba4203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.035756 28604 raft_consensus.cc:348] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.080902 28604 raft_consensus.cc:374] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.080996 28604 raft_consensus.cc:729] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.081204 28604 consensus_queue.cc:260] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.082392 28604 ts_tablet_manager.cc:1398] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.048s user 0.001s sys 0.002s
I20240503 14:12:21.083154 28604 tablet_bootstrap.cc:492] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:21.084427 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 5611fbd499cb449b929973facb2d914e (DEFAULT_TABLE table=default.Table_178cd961b2bf465596a81dd8833ea327 [id=bf6caff271654ddbbeb846bb171b7a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.084829 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5611fbd499cb449b929973facb2d914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.089206 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 5d308130f0dc4c16ac73058db332b41d (DEFAULT_TABLE table=default.Table_20718ac286c9405b871c43b98ac94f8d [id=51930b6b93bd406581a479188de3423d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.089877 28191 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc9815b8d349f2ac4b6d380d369638 (DEFAULT_TABLE table=default.Table_c0420d9a80af4738972115df9798052c [id=51f8df4cf4ae4790b97820f8f8f845d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.090454 28182 tablet_service.cc:1467] Processing CreateTablet for tablet c92b0eec16a64dcdb0a5bee2a030e507 (DEFAULT_TABLE table=default.Table_e1abe3457b4248fb826e4c21037ceb83 [id=eedbf1330b194490a0d7c97e44b4216d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.091079 28186 tablet_service.cc:1467] Processing CreateTablet for tablet a53279669bad40ad9243f4554a4e3c70 (DEFAULT_TABLE table=default.Table_a65e77322444449cadf78698f2c98fd6 [id=b4e3102125404739b79234c6c7d1c7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.091662 28195 tablet_service.cc:1467] Processing CreateTablet for tablet af0cf232932d48f880f7aced9c38b714 (DEFAULT_TABLE table=default.Table_2aeaeda9485f4eb1953af7e447a714ce [id=592270a7c17247eabc9fbb9195bb1a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.091794 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a92fc3833e49aaab8c5415de9bb685" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.092402 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495190fca06a40a386c8215eb2195920" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.091806 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 244718a7d40d4ad0ac7c3a41bdb325d3 (DEFAULT_TABLE table=default.Table_6aea0eb4f439481da1da149c63f49de8 [id=32e1a0d216794a1085930a2cf694f88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.097350 28653 tablet_replica.cc:330] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:21.097680 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 2a691ceaa2b3464d83b722027ced0a96 (DEFAULT_TABLE table=default.Table_50e8eea84f564a4da088dd0efa136f69 [id=b996d4c5211c43b28d527f6b7501e99e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.098048 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1f3443c298426db9ff5df1437884f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.100890 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8c51c0baf44430894429f6eaea797f (DEFAULT_TABLE table=default.Table_2cb9f4b3da44486fbb7fdbf277833634 [id=1aa5e872eb3846998d2fd9c1a511470e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.101169 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet cf221f17ddfd461f83f7f916d7833011 with delete_type TABLET_DATA_DELETED (replaced by tablet 5de9edfb0d064939bda2ec7a9e0e500b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.101523 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92b0eec16a64dcdb0a5bee2a030e507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.101550 28600 tablet_bootstrap.cc:654] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.102620 29845 raft_consensus.cc:482] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.103404 29845 raft_consensus.cc:504] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.106412 28326 tablet_service.cc:1467] Processing CreateTablet for tablet aad397ed76ea4cf8820312ba1362512f (DEFAULT_TABLE table=default.Table_435f8c6696444d43a33900aee0682c9a [id=dfc716c4d90e411e963b25a32772debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.107421 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad397ed76ea4cf8820312ba1362512f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.107606 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc9815b8d349f2ac4b6d380d369638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.107923 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0cf232932d48f880f7aced9c38b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.109163 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 0c300b1f104949b3ae74f5b38d378a11 (DEFAULT_TABLE table=default.Table_10dee84bef6148bf8704fd1da1e39b58 [id=7091b198d9a940cb8132bb92f5084114]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.110447 28212 raft_consensus.cc:2454] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:21.111354 28653 raft_consensus.cc:2229] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.111460 28213 raft_consensus.cc:2454] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:21.112217 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53279669bad40ad9243f4554a4e3c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.112303 28653 raft_consensus.cc:2258] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.113615 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353d55280fac4953a54347c3756adcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.114710 28214 raft_consensus.cc:2454] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:21.114763 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.115520 28653 ts_tablet_manager.cc:1875] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.115571 28188 tablet_service.cc:1467] Processing CreateTablet for tablet a1f53b98b55349a8ab937f58662b679f (DEFAULT_TABLE table=default.Table_b530238524784f869cd09190d81b9726 [id=cb87eb1b4c8c4d2487dae28d11b81b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.115866 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d308130f0dc4c16ac73058db332b41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.114387 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f89fca0480d4bf28160384ae0421bce" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.116385 28215 raft_consensus.cc:2454] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:21.132970 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409cbf0cb8ef44819cfdb36f2fe83110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.134485 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a691ceaa2b3464d83b722027ced0a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.134912 28194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf8f068f8424317bbfce6ff10bba61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.138077 28331 tablet_service.cc:1467] Processing CreateTablet for tablet b04fff656f624001a709f783816eec1e (DEFAULT_TABLE table=default.Table_9aebe1270f6749608f4ec392cecede93 [id=b4345d6ea1e2403b8e851145944d40e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.138604 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04fff656f624001a709f783816eec1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.141146 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5b973ea16d42babd2cc75e32624854 (DEFAULT_TABLE table=default.Table_1fafcfc2a1b747f0a7047b61caefb5df [id=ce80aae3983c43dfbc6786dbcf5849e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.141592 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5b973ea16d42babd2cc75e32624854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.143131 28334 tablet_service.cc:1467] Processing CreateTablet for tablet 429cbd2310c940ad9a8d7a4f4ef17b3b (DEFAULT_TABLE table=default.Table_5ba631c9bf2f4374991bda5d67c18af0 [id=9de6b0707c114f0ebed19048f7b85833]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.143594 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 429cbd2310c940ad9a8d7a4f4ef17b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.145970 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf0a1f98bf14aa48fc5b5a805716309 (DEFAULT_TABLE table=default.Table_006d60d6900a406b8a5c73a795718f63 [id=1a5a92f8c6d04eca9b4267fae3e016ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.146376 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf0a1f98bf14aa48fc5b5a805716309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.152288 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244718a7d40d4ad0ac7c3a41bdb325d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.153807 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad21d284e794156a5b816897d807a50 (DEFAULT_TABLE table=default.Table_1ef8c263cd444fe989ea219bba7f5829 [id=7dd241787ce54ce6be06f136968cfd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.154346 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad21d284e794156a5b816897d807a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.154263 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 005599a6f11843aca07307eaf91e7924 (DEFAULT_TABLE table=default.Table_34131ceab94648048a945b75d3042a55 [id=ccd353dbb8f94ac1af8a109991e98aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.154798 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005599a6f11843aca07307eaf91e7924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.159832 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb6d931156b492e82140ee342cb1bbb (DEFAULT_TABLE table=default.Table_92abfaed91ab4683a6b2e501ce673e58 [id=6539aa29cb864a2fa19c8a2f1ac7bd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.160349 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb6d931156b492e82140ee342cb1bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.164969 29845 leader_election.cc:288] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.165726 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 31721f852044499b84ad17c4b31964a4 (DEFAULT_TABLE table=default.Table_521b7e40755f4748ad55fc13b690b642 [id=2cf40af77c234dae91434a6034d7d51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.165773 28323 tablet_service.cc:1467] Processing CreateTablet for tablet da204b27173840f4b0ae09166dabbe13 (DEFAULT_TABLE table=default.Table_b676719e70fe4322adac2fa87c4cbd1b [id=62a36d8d3e3d4152a9b442daacca2a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.166201 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31721f852044499b84ad17c4b31964a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.168488 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da204b27173840f4b0ae09166dabbe13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.169430 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 5de9edfb0d064939bda2ec7a9e0e500b (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.169904 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de9edfb0d064939bda2ec7a9e0e500b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.171003 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 140de4d395d54b7696062ed96b8aa89c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.171324 29828 tablet_replica.cc:330] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.171324 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 903f11a608b1483887d9b5234a404470 (table default.Table_4afe0b312f964d3885779a752ec8c256 [id=32be054076954aed9265888ca62e510b]) successfully deleted
I20240503 14:12:21.171066 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.171916 28327 tablet_service.cc:1467] Processing CreateTablet for tablet d952b436198141ad80fef5fb1ea1e736 (DEFAULT_TABLE table=default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea [id=23231f74e45e4d3484db472d381f97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.172338 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d952b436198141ad80fef5fb1ea1e736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.183683 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b8c51c0baf44430894429f6eaea797f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:21.184800 28290 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 903f11a608b1483887d9b5234a404470
I20240503 14:12:21.184947 28290 leader_election.cc:302] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:21.185643 28290 leader_election.cc:341] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 95d9985eecda4f3194c3251216aa58ee
I20240503 14:12:21.185767 28290 leader_election.cc:302] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:21.190811 29773 raft_consensus.cc:482] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.191151 29773 raft_consensus.cc:504] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.192485 29773 leader_election.cc:288] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.192809 29773 raft_consensus.cc:2726] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.193104 29773 raft_consensus.cc:2726] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.200409 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f53b98b55349a8ab937f58662b679f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.204221 29845 raft_consensus.cc:482] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.204424 29845 raft_consensus.cc:504] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.205166 29845 leader_election.cc:288] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.206027 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet a53279669bad40ad9243f4554a4e3c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.207418 28614 tablet_bootstrap.cc:492] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.207688 28614 ts_tablet_manager.cc:1367] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.192s user 0.009s sys 0.013s
I20240503 14:12:21.208541 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c300b1f104949b3ae74f5b38d378a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.209717 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.210311 28182 tablet_service.cc:1467] Processing CreateTablet for tablet 3366f813bd1f4df3843487ad40ba4203 (DEFAULT_TABLE table=default.Table_f652e382ed2b40da8e5677659ce13416 [id=c07d28d716c14843b03e915b3a3023c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.210856 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3366f813bd1f4df3843487ad40ba4203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.220796 28614 raft_consensus.cc:348] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.221136 28614 raft_consensus.cc:374] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.221235 28614 raft_consensus.cc:729] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.221705 28614 consensus_queue.cc:260] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.222353 28614 ts_tablet_manager.cc:1398] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:12:21.215139 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11137b75794e2c9dd61ded326c868f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:21.222613 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 86804987bdf340dc83ddd58ba2bc8a96 (table default.Table_d97ddf0257014cd7835a63e1730e6907 [id=9c06c541eab341558ee262552c52b522], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 86804987bdf340dc83ddd58ba2bc8a96 consensus state has no leader
I20240503 14:12:21.240136 28614 tablet_bootstrap.cc:492] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.243682 29874 tablet.cc:1722] T f2b861dccd964092ae0f06668c9b58fe P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:21.243820 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b8d8f09130c04698949adc45f8b071d7
I20240503 14:12:21.247190 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c64ecffdd4a84ba5bc64aba80e1de1ab" } new_table_name: "default.Table_40b730849b5c46d3bbbb0e15cf2f95d5" modify_external_catalogs: true
I20240503 14:12:21.254724 29841 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:12:21.255015 29510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38038 (request call id 0) took 5129 ms. Trace:
I20240503 14:12:21.255131 29841 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:12:21.255544 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 903f11a608b1483887d9b5234a404470 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e7b2865e9604ce58280321c7ee2a88e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.256103 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 5611fbd499cb449b929973facb2d914e (DEFAULT_TABLE table=default.Table_178cd961b2bf465596a81dd8833ea327 [id=bf6caff271654ddbbeb846bb171b7a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.256700 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5611fbd499cb449b929973facb2d914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.263151 29845 raft_consensus.cc:482] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.263419 29845 raft_consensus.cc:504] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.264273 29845 leader_election.cc:288] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.264891 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 3366f813bd1f4df3843487ad40ba4203 (DEFAULT_TABLE table=default.Table_f652e382ed2b40da8e5677659ce13416 [id=c07d28d716c14843b03e915b3a3023c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.265014 28614 tablet_bootstrap.cc:654] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.247718 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 37dac592c9dd4888a2ce89fa922f7fc8
I20240503 14:12:21.057742 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903f11a608b1483887d9b5234a404470" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:21.265571 28355 raft_consensus.cc:2379] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:21.265692 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 495190fca06a40a386c8215eb2195920 with delete_type TABLET_DATA_DELETED (replaced by tablet 7261088abbc641db8d737a48bbac25bd) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.265952 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3366f813bd1f4df3843487ad40ba4203. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:21.248134 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bab214b7ec58425d8cd1180ee16b273d consensus state has no leader
I20240503 14:12:21.268716 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_717ce9be26654b1d9f243797798e0fa3" } new_table_name: "default.Table_79f99b72ae704c34a5a58eaac219d32e" modify_external_catalogs: true
I20240503 14:12:21.250706 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_664e113b7fe34816ad9b995b341c40ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.251199 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8097ba5606cf42d68448833de36d85c7
I20240503 14:12:21.278270 28033 catalog_manager.cc:6554] ReplaceTablet: tablet 8097ba5606cf42d68448833de36d85c7 deleted and replaced by tablet c82c2edb1809410c9f11ef639e723db6
I20240503 14:12:21.278858 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7e8c0ee351544a32a03b6aa78c2f9264"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.279718 28472 tablet_service.cc:1467] Processing CreateTablet for tablet b04fff656f624001a709f783816eec1e (DEFAULT_TABLE table=default.Table_9aebe1270f6749608f4ec392cecede93 [id=b4345d6ea1e2403b8e851145944d40e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.271996 29803 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0690a516c89446d85c88cdf64fa4238. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:21.280215 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04fff656f624001a709f783816eec1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.251936 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c6d80a93009d467ab3bc0bc3c3ba529f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.252463 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 927d4359637f4ec189a5143e8201c7f1
I20240503 14:12:21.282480 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fc8c9ef982c344d697bc3293363f14d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.253852 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet e6c90869b5034128864faaf05697c230 (table default.Table_5dcdff74f1f7407084a45dc9fe588103 [id=47ade8624b00424d8fc04dbca5174ff4]). Expected version 3 got 0
I20240503 14:12:21.283950 28461 tablet_service.cc:1467] Processing CreateTablet for tablet aad397ed76ea4cf8820312ba1362512f (DEFAULT_TABLE table=default.Table_435f8c6696444d43a33900aee0682c9a [id=dfc716c4d90e411e963b25a32772debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.264523 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e8f9f10348014293a421ed11a50f89e7
I20240503 14:12:21.284432 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad397ed76ea4cf8820312ba1362512f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.284869 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_41c48bdfa525455ea062c1d933c14447" } new_table_name: "default.Table_b4868a6959c44bfbaad9b187dd171f41" modify_external_catalogs: true
I20240503 14:12:21.290609 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f526ebddbfd04608bdf15d7af168e12a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.293638 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91eb4433685b4119a170054a0a777e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.295841 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 5611fbd499cb449b929973facb2d914e (DEFAULT_TABLE table=default.Table_178cd961b2bf465596a81dd8833ea327 [id=bf6caff271654ddbbeb846bb171b7a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.300338 29872 raft_consensus.cc:482] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.300521 29872 raft_consensus.cc:504] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.301302 29872 leader_election.cc:288] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:21.306187 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5611fbd499cb449b929973facb2d914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.307026 28186 tablet_service.cc:1467] Processing CreateTablet for tablet aad397ed76ea4cf8820312ba1362512f (DEFAULT_TABLE table=default.Table_435f8c6696444d43a33900aee0682c9a [id=dfc716c4d90e411e963b25a32772debd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:21.311522 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6d80a93009d467ab3bc0bc3c3ba529f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:21.313213 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc8c9ef982c344d697bc3293363f14d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:21.315112 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e8c0ee351544a32a03b6aa78c2f9264 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:21.317140 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_664e113b7fe34816ad9b995b341c40ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:21.318857 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f526ebddbfd04608bdf15d7af168e12a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.323114 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d1314841574fc8bcf3ccb16755e421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.326799 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_decc999a1649405b8e7301584192c536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.327373 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5b973ea16d42babd2cc75e32624854 (DEFAULT_TABLE table=default.Table_1fafcfc2a1b747f0a7047b61caefb5df [id=ce80aae3983c43dfbc6786dbcf5849e5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:21.327517 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_decc999a1649405b8e7301584192c536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.331974 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3112560e16f14b4e89aae90c014be9fb
W20240503 14:12:21.334069 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 148420us
I20240503 14:12:21.339139 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5b973ea16d42babd2cc75e32624854 (DEFAULT_TABLE table=default.Table_1fafcfc2a1b747f0a7047b61caefb5df [id=ce80aae3983c43dfbc6786dbcf5849e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.339231 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 429cbd2310c940ad9a8d7a4f4ef17b3b (DEFAULT_TABLE table=default.Table_5ba631c9bf2f4374991bda5d67c18af0 [id=9de6b0707c114f0ebed19048f7b85833]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.339641 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5b973ea16d42babd2cc75e32624854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.339944 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 429cbd2310c940ad9a8d7a4f4ef17b3b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:21.341485 28290 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 903f11a608b1483887d9b5234a404470
I20240503 14:12:21.342561 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_463c966bd4c847d0a0967bc7dae04ec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.344568 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_463c966bd4c847d0a0967bc7dae04ec6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:21.347242 28290 consensus_peers.cc:485] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 140de4d395d54b7696062ed96b8aa89c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:21.348493 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c5dfac02d4dfe932996d2a37fe031" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.349015 28491 raft_consensus.cc:2379] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:21.354516 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 005599a6f11843aca07307eaf91e7924 (DEFAULT_TABLE table=default.Table_34131ceab94648048a945b75d3042a55 [id=ccd353dbb8f94ac1af8a109991e98aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.354609 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad21d284e794156a5b816897d807a50 (DEFAULT_TABLE table=default.Table_1ef8c263cd444fe989ea219bba7f5829 [id=7dd241787ce54ce6be06f136968cfd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.355000 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005599a6f11843aca07307eaf91e7924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.355515 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad21d284e794156a5b816897d807a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.356611 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 8097ba5606cf42d68448833de36d85c7 with delete_type TABLET_DATA_DELETED (replaced by tablet c82c2edb1809410c9f11ef639e723db6) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.356880 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d8f09130c04698949adc45f8b071d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.356990 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 927d4359637f4ec189a5143e8201c7f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.357102 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dac592c9dd4888a2ce89fa922f7fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.357442 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abcb8658fc34c939aa74cf0bd981ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.357479 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f9f10348014293a421ed11a50f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.357843 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 60206e8e73e84e409ed6aba222d9ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.358134 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7c5dfac02d4dfe932996d2a37fe031 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d45788ae644a8581acca54cc55dd3f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.358407 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 903f11a608b1483887d9b5234a404470 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e7b2865e9604ce58280321c7ee2a88e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.358670 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d9985eecda4f3194c3251216aa58ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.358909 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 3112560e16f14b4e89aae90c014be9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.360702 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf0a1f98bf14aa48fc5b5a805716309 (DEFAULT_TABLE table=default.Table_006d60d6900a406b8a5c73a795718f63 [id=1a5a92f8c6d04eca9b4267fae3e016ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.361150 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf0a1f98bf14aa48fc5b5a805716309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.361084 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.361371 28493 raft_consensus.cc:2379] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:21.369298 29828 raft_consensus.cc:2229] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:21.370072 29828 raft_consensus.cc:2258] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.371151 29828 ts_tablet_manager.cc:1875] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.371439 28604 tablet_bootstrap.cc:654] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.377508 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 5de9edfb0d064939bda2ec7a9e0e500b (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.377971 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de9edfb0d064939bda2ec7a9e0e500b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.379199 29828 ts_tablet_manager.cc:1888] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:21.379335 29828 log.cc:1196] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/140de4d395d54b7696062ed96b8aa89c
I20240503 14:12:21.379706 29828 ts_tablet_manager.cc:1909] T 140de4d395d54b7696062ed96b8aa89c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.380290 28614 tablet_bootstrap.cc:492] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.380442 28614 ts_tablet_manager.cc:1367] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.140s user 0.006s sys 0.000s
I20240503 14:12:21.381021 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb6d931156b492e82140ee342cb1bbb (DEFAULT_TABLE table=default.Table_92abfaed91ab4683a6b2e501ce673e58 [id=6539aa29cb864a2fa19c8a2f1ac7bd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.381321 29828 tablet_replica.cc:330] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.381366 28614 raft_consensus.cc:348] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.381536 29828 raft_consensus.cc:2229] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:21.381582 28614 raft_consensus.cc:374] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.381634 28614 raft_consensus.cc:729] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.381807 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb6d931156b492e82140ee342cb1bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.381852 28614 consensus_queue.cc:260] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.386400 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 31721f852044499b84ad17c4b31964a4 (DEFAULT_TABLE table=default.Table_521b7e40755f4748ad55fc13b690b642 [id=2cf40af77c234dae91434a6034d7d51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.386853 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31721f852044499b84ad17c4b31964a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.293560 28661 ts_tablet_manager.cc:1888] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.392495 28661 log.cc:1196] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/95d9985eecda4f3194c3251216aa58ee
I20240503 14:12:21.392977 28661 ts_tablet_manager.cc:1909] T 95d9985eecda4f3194c3251216aa58ee P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:21.394281 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 725) took 3169 ms. Trace:
I20240503 14:12:21.394467 28661 rpcz_store.cc:276] 0503 14:12:18.225308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.225360 (+ 52us) service_pool.cc:225] Handling call
0503 14:12:20.812497 (+2587137us) ts_tablet_manager.cc:1120] Deleting tablet 95d9985eecda4f3194c3251216aa58ee
0503 14:12:20.812507 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.063951 (+251444us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.293418 (+229467us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.394255 (+100837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2586991}
I20240503 14:12:21.395259 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 727) took 3121 ms. Trace:
I20240503 14:12:21.395428 28661 rpcz_store.cc:276] 0503 14:12:18.274193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.281464 (+ 7271us) service_pool.cc:225] Handling call
0503 14:12:21.395086 (+3113622us) ts_tablet_manager.cc:1120] Deleting tablet cf810e84367f420b928a171790849445
0503 14:12:21.395097 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.395239 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3112497}
I20240503 14:12:21.395906 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 729) took 3009 ms. Trace:
I20240503 14:12:21.396039 28661 rpcz_store.cc:276] 0503 14:12:18.386051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.422526 (+ 36475us) service_pool.cc:225] Handling call
0503 14:12:21.395785 (+2973259us) ts_tablet_manager.cc:1120] Deleting tablet fc3f7c221cbc467dabfe5665ccd07d48
0503 14:12:21.395795 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.395889 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2972651}
I20240503 14:12:21.396428 28661 tablet_replica.cc:330] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:21.396679 28661 raft_consensus.cc:2229] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:21.397802 28661 pending_rounds.cc:62] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:21.414703 29828 pending_rounds.cc:62] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:21.414908 29864 raft_consensus.cc:482] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.415091 28614 ts_tablet_manager.cc:1398] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240503 14:12:21.415092 29864 raft_consensus.cc:504] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.415421 28614 tablet_bootstrap.cc:492] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.415951 29864 leader_election.cc:288] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:21.417479 28614 tablet_bootstrap.cc:654] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.414891 29828 pending_rounds.cc:68] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597733646962688 op_type: NO_OP noop_request { }
I20240503 14:12:21.419863 29828 raft_consensus.cc:2866] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:21.419983 29828 raft_consensus.cc:2258] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.421114 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 140de4d395d54b7696062ed96b8aa89c (table default.Table_dbb3e0a033c84560a6458571881b25e5 [id=238df3dccaa54f4f956476ccb58b0816]) successfully deleted
I20240503 14:12:21.421164 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf0a1f98bf14aa48fc5b5a805716309 (DEFAULT_TABLE table=default.Table_006d60d6900a406b8a5c73a795718f63 [id=1a5a92f8c6d04eca9b4267fae3e016ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.421710 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf0a1f98bf14aa48fc5b5a805716309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.424578 29828 ts_tablet_manager.cc:1875] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:21.436228 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9028a6990d0840ffa6849143f6c004db consensus state has no leader
W20240503 14:12:21.448320 28290 leader_election.cc:341] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6c7c5dfac02d4dfe932996d2a37fe031
I20240503 14:12:21.448596 28290 leader_election.cc:302] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:21.449045 29864 raft_consensus.cc:3037] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.279995 29803 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:21.299623 29841 leader_election.cc:288] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:12:21.452282 29864 raft_consensus.cc:2726] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.431844 29486 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" instance_seqno: 1714745518759808) as {username='slave'} at 127.27.50.194:48633; Asking this server to re-register.
I20240503 14:12:21.426041 29876 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20240503 14:12:21.456727 28290 leader_election.cc:341] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 60206e8e73e84e409ed6aba222d9ff4b
I20240503 14:12:21.464213 29828 ts_tablet_manager.cc:1888] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:21.464397 28406 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:21.467654 28614 tablet_bootstrap.cc:492] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.467830 28614 ts_tablet_manager.cc:1367] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.005s
I20240503 14:12:21.469765 28614 raft_consensus.cc:348] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.470196 28614 raft_consensus.cc:374] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.470268 28614 raft_consensus.cc:729] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.470697 28614 consensus_queue.cc:260] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.472229 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_70df8ea2254f45aab971123dd7f55a44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.472705 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4db31dc8ab934c1595422943e50fe9c8" } new_table_name: "default.Table_66fe7131976648bf8170857505ebb84e" modify_external_catalogs: true
W20240503 14:12:21.472895 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70df8ea2254f45aab971123dd7f55a44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:21.474323 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c29aad1eee3541c5b483b193d769a8da consensus state has no leader
I20240503 14:12:21.474704 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet 10876f4512794e6aa72bda3c1da8b2d0 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:21.475363 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:21.476006 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1b9a58dce8ab4927bd0bd665004b2e3b" } modify_external_catalogs: true
I20240503 14:12:21.476311 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1b9a58dce8ab4927bd0bd665004b2e3b" } modify_external_catalogs: true
I20240503 14:12:21.478531 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_39fec5701a7a485caa43e8694f08bbf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.479121 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39fec5701a7a485caa43e8694f08bbf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.481096 28614 ts_tablet_manager.cc:1398] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:12:21.481366 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 920b41e3b21347b393497de4d4e87469 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:21.481586 28614 tablet_bootstrap.cc:492] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.482017 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 752b369f24224f2486834814a9d08495 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:21.483588 28614 tablet_bootstrap.cc:654] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.483736 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0d6cc797740e4f18a7aad22805859cd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.484647 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d6cc797740e4f18a7aad22805859cd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.485996 29867 raft_consensus.cc:482] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.486150 29867 raft_consensus.cc:504] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.486894 29867 leader_election.cc:288] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:21.488723 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_89b0f6a84c3140aca776c1da1137e09f" } modify_external_catalogs: true
I20240503 14:12:21.489046 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_89b0f6a84c3140aca776c1da1137e09f" } modify_external_catalogs: true
W20240503 14:12:21.489723 28290 leader_election.cc:341] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 7abcb8658fc34c939aa74cf0bd981ae4
I20240503 14:12:21.490134 29828 log.cc:1196] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/1c5e3649c71f4d4da710c69f85a82c25
I20240503 14:12:21.490620 29828 ts_tablet_manager.cc:1909] T 1c5e3649c71f4d4da710c69f85a82c25 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.492422 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 1c5e3649c71f4d4da710c69f85a82c25 (table default.Table_dc507e002d97466bae378a75112560a8 [id=3a0db192d94f40c3b69f089100ad18ae]) successfully deleted
W20240503 14:12:21.492749 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 8097ba5606cf42d68448833de36d85c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8097ba5606cf42d68448833de36d85c7 already in progress: creating tablet
W20240503 14:12:21.492992 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b8d8f09130c04698949adc45f8b071d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d8f09130c04698949adc45f8b071d7 already in progress: creating tablet
W20240503 14:12:21.493228 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 927d4359637f4ec189a5143e8201c7f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 927d4359637f4ec189a5143e8201c7f1 already in progress: creating tablet
W20240503 14:12:21.493480 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 37dac592c9dd4888a2ce89fa922f7fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37dac592c9dd4888a2ce89fa922f7fc8 already in progress: creating tablet
I20240503 14:12:21.499341 29828 tablet_replica.cc:330] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.499644 29828 raft_consensus.cc:2229] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.499903 29828 raft_consensus.cc:2258] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.502853 29828 ts_tablet_manager.cc:1875] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.507165 29486 ts_manager.cc:194] Registered new tserver with Master: d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:21.510023 29828 ts_tablet_manager.cc:1888] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.510295 29828 log.cc:1196] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7abcb8658fc34c939aa74cf0bd981ae4
I20240503 14:12:21.510699 29828 ts_tablet_manager.cc:1909] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.511588 29828 tablet_replica.cc:330] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.511960 29828 raft_consensus.cc:2229] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.512372 29828 raft_consensus.cc:2258] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.512578 28614 tablet_bootstrap.cc:492] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.512723 28614 ts_tablet_manager.cc:1367] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20240503 14:12:21.513685 28614 raft_consensus.cc:348] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.513902 28614 raft_consensus.cc:374] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.513849 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7abcb8658fc34c939aa74cf0bd981ae4 (table default.Table_80c5e40f9e494b2dbbbb7449a5b36569 [id=07297b58e7ca4fb1bf0e050a341d2c9c]) successfully deleted
I20240503 14:12:21.513957 28614 raft_consensus.cc:729] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.513821 29828 ts_tablet_manager.cc:1875] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.514241 28614 consensus_queue.cc:260] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:21.514477 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e8f9f10348014293a421ed11a50f89e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8f9f10348014293a421ed11a50f89e7 already in progress: creating tablet
I20240503 14:12:21.514827 28614 ts_tablet_manager.cc:1398] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:21.515091 28614 tablet_bootstrap.cc:492] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.517019 28614 tablet_bootstrap.cc:654] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.521771 28900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 802 } } ignore_live_leader: false dest_uuid: "bb6a7aee99214df4b02858ddae025b14" is_pre_election: true
I20240503 14:12:21.527266 29828 ts_tablet_manager.cc:1888] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.527626 29828 log.cc:1196] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/60206e8e73e84e409ed6aba222d9ff4b
I20240503 14:12:21.528188 29828 ts_tablet_manager.cc:1909] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.529081 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 60206e8e73e84e409ed6aba222d9ff4b (table default.Table_e19b96f79a6c41c0bc266f1391941564 [id=c7144a78f2ae47599016aa64b6d0cbea]) successfully deleted
I20240503 14:12:21.529286 29828 tablet_replica.cc:330] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.529636 29828 raft_consensus.cc:2229] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.529985 29828 raft_consensus.cc:2258] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.533911 28027 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_53de0d5698bd46a3869b5d8c8a1a7233 [id=aaeb9d44d9f04208a57d49b81068662d] alter complete (version 2)
I20240503 14:12:21.534966 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_777d8101e5ba4707a324b1ea2b061d2a" } new_table_name: "default.Table_5654996201694c1b90fc6e728e068ca3" modify_external_catalogs: true
W20240503 14:12:21.535220 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95a7da6f939b432da3986e000406a3ad consensus state has no leader
I20240503 14:12:21.535706 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_04ae17bd98c04c4b993d808882cdc3ec" } modify_external_catalogs: true
I20240503 14:12:21.535917 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_04ae17bd98c04c4b993d808882cdc3ec" } modify_external_catalogs: true
I20240503 14:12:21.538605 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.539422 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea" } modify_external_catalogs: true
I20240503 14:12:21.539654 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea" } modify_external_catalogs: true
I20240503 14:12:21.547076 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]). Expected version 3 got 0
I20240503 14:12:21.548638 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7cafc45f1ec54604b2608f91acef4847" } modify_external_catalogs: true
I20240503 14:12:21.548921 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7cafc45f1ec54604b2608f91acef4847" } modify_external_catalogs: true
I20240503 14:12:21.554451 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 44b8cc9a520a4ab38cd53fb2ab9789b5 (DEFAULT_TABLE table=default.Table_d8070970f59847d091fc2692c79d2ae9 [id=aefbbf7fce604654a8b6f8ca2e872f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.554956 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b8cc9a520a4ab38cd53fb2ab9789b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.555069 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a081ba59ad44890bf2f94874c034277 with delete_type TABLET_DATA_DELETED (replaced by tablet d859044b70fc48e8b62bf6a82ed69d9f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.556041 29828 ts_tablet_manager.cc:1875] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.560436 28325 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 with delete_type TABLET_DATA_DELETED (replaced by tablet 70f0e07d32084999a3ff42f69bb1ca2f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.399507 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d8f09130c04698949adc45f8b071d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.399706 28456 tablet_service.cc:1467] Processing CreateTablet for tablet d952b436198141ad80fef5fb1ea1e736 (DEFAULT_TABLE table=default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea [id=23231f74e45e4d3484db472d381f97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.561118 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d952b436198141ad80fef5fb1ea1e736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.562597 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 7e644199208a45db99c085cd356a0584 (DEFAULT_TABLE table=default.Table_3838859d5d884075bc288faedfd9ce98 [id=a683c6d18deb4968ade55b5edd730f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.563038 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e644199208a45db99c085cd356a0584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.402159 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dac592c9dd4888a2ce89fa922f7fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.402691 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 8097ba5606cf42d68448833de36d85c7 with delete_type TABLET_DATA_DELETED (replaced by tablet c82c2edb1809410c9f11ef639e723db6) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.403466 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet 927d4359637f4ec189a5143e8201c7f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.404124 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f9f10348014293a421ed11a50f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.404681 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a081ba59ad44890bf2f94874c034277 with delete_type TABLET_DATA_DELETED (replaced by tablet d859044b70fc48e8b62bf6a82ed69d9f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.405261 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 3112560e16f14b4e89aae90c014be9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.416800 29881 tablet.cc:1722] T f2b861dccd964092ae0f06668c9b58fe P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:21.421890 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60206e8e73e84e409ed6aba222d9ff4b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.569660 28489 raft_consensus.cc:2454] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:21.570446 28290 leader_election.cc:302] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:21.570580 28614 tablet_bootstrap.cc:492] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.570816 28614 ts_tablet_manager.cc:1367] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.000s
W20240503 14:12:21.570840 28290 raft_consensus.cc:2670] T 60206e8e73e84e409ed6aba222d9ff4b P d3a6b9b7913444d29c96581af6febf33: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240503 14:12:21.486091 28427 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 80621.3us
I20240503 14:12:21.571712 28614 raft_consensus.cc:348] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.571936 28614 raft_consensus.cc:374] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.572002 28614 raft_consensus.cc:729] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.489405 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abcb8658fc34c939aa74cf0bd981ae4" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.572260 28491 raft_consensus.cc:2454] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:21.572289 28614 consensus_queue.cc:260] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.573249 28614 ts_tablet_manager.cc:1398] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:21.493300 28604 tablet_bootstrap.cc:492] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:21.576617 28604 ts_tablet_manager.cc:1367] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.494s user 0.004s sys 0.001s
I20240503 14:12:21.577623 28604 raft_consensus.cc:348] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.577960 28604 raft_consensus.cc:374] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.578158 28604 raft_consensus.cc:729] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.578481 28604 consensus_queue.cc:260] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.579305 28604 ts_tablet_manager.cc:1398] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:21.579252 28290 leader_election.cc:302] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
W20240503 14:12:21.579468 28290 raft_consensus.cc:2670] T 7abcb8658fc34c939aa74cf0bd981ae4 P d3a6b9b7913444d29c96581af6febf33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:21.579687 28604 tablet_bootstrap.cc:492] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:21.508523 28473 tablet_service.cc:1467] Processing CreateTablet for tablet da204b27173840f4b0ae09166dabbe13 (DEFAULT_TABLE table=default.Table_b676719e70fe4322adac2fa87c4cbd1b [id=62a36d8d3e3d4152a9b442daacca2a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.583218 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 95d9985eecda4f3194c3251216aa58ee (table default.Table_627b41e76476426da74d6722301bc59f [id=97e7cb47edc44a5e9041be7ad4bc6f47]) successfully deleted
I20240503 14:12:21.583745 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da204b27173840f4b0ae09166dabbe13. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:21.583750 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet cf810e84367f420b928a171790849445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf810e84367f420b928a171790849445 already in progress: creating tablet
I20240503 14:12:21.584976 29845 raft_consensus.cc:482] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.585126 29845 raft_consensus.cc:504] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:21.585047 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet fc3f7c221cbc467dabfe5665ccd07d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc3f7c221cbc467dabfe5665ccd07d48 already in progress: creating tablet
I20240503 14:12:21.586443 29845 leader_election.cc:288] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.586510 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:21.586694 28661 pending_rounds.cc:68] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597725382160384 op_type: NO_OP noop_request { }
I20240503 14:12:21.586918 28661 raft_consensus.cc:2866] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:21.587118 28661 raft_consensus.cc:2258] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.591629 28325 tablet_service.cc:1467] Processing CreateTablet for tablet cce4c617ee694c36b3e63d51d6e0f209 (DEFAULT_TABLE table=default.Table_d039e7aa896743fe87bb9365dfe8a704 [id=896cb6b2e8b74e16960f3c6577ed8ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.592587 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.592969 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf21f6234dd64cf8a32f1599fe50746e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.593269 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.594257 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce4c617ee694c36b3e63d51d6e0f209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.594483 28661 ts_tablet_manager.cc:1875] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.594782 28614 tablet_bootstrap.cc:492] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.596877 28614 tablet_bootstrap.cc:654] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.599439 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 802 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
W20240503 14:12:21.599843 28427 leader_election.cc:341] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 7abcb8658fc34c939aa74cf0bd981ae4
I20240503 14:12:21.599970 28427 leader_election.cc:302] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.608139 28614 tablet_bootstrap.cc:492] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.608305 28614 ts_tablet_manager.cc:1367] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240503 14:12:21.609164 28614 raft_consensus.cc:348] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.609429 28614 raft_consensus.cc:374] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.609488 28614 raft_consensus.cc:729] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.609750 28614 consensus_queue.cc:260] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.610489 28614 ts_tablet_manager.cc:1398] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:21.610805 28614 tablet_bootstrap.cc:492] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.432471 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad21d284e794156a5b816897d807a50 (DEFAULT_TABLE table=default.Table_1ef8c263cd444fe989ea219bba7f5829 [id=7dd241787ce54ce6be06f136968cfd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.435401 28180 tablet_service.cc:1467] Processing CreateTablet for tablet b04fff656f624001a709f783816eec1e (DEFAULT_TABLE table=default.Table_9aebe1270f6749608f4ec392cecede93 [id=b4345d6ea1e2403b8e851145944d40e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.442668 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c90869b5034128864faaf05697c230" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.612830 28614 tablet_bootstrap.cc:654] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.472627 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 429cbd2310c940ad9a8d7a4f4ef17b3b (DEFAULT_TABLE table=default.Table_5ba631c9bf2f4374991bda5d67c18af0 [id=9de6b0707c114f0ebed19048f7b85833]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.476687 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb6d931156b492e82140ee342cb1bbb (DEFAULT_TABLE table=default.Table_92abfaed91ab4683a6b2e501ce673e58 [id=6539aa29cb864a2fa19c8a2f1ac7bd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.476873 28185 tablet_service.cc:1467] Processing CreateTablet for tablet da204b27173840f4b0ae09166dabbe13 (DEFAULT_TABLE table=default.Table_b676719e70fe4322adac2fa87c4cbd1b [id=62a36d8d3e3d4152a9b442daacca2a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.480207 28189 tablet_service.cc:1467] Processing CreateTablet for tablet 5de9edfb0d064939bda2ec7a9e0e500b (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:21.480990 28150 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.007s sys 0.012s
W20240503 14:12:21.613907 28150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s user 0.007s sys 0.012s
I20240503 14:12:21.613740 29437 leader_election.cc:302] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc; no voters: bb6a7aee99214df4b02858ddae025b14, c0690a516c89446d85c88cdf64fa4238
W20240503 14:12:21.614109 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 794988us
I20240503 14:12:21.614308 28290 leader_election.cc:302] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:21.614615 29867 raft_consensus.cc:3037] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.481444 28178 tablet_service.cc:1467] Processing CreateTablet for tablet d952b436198141ad80fef5fb1ea1e736 (DEFAULT_TABLE table=default.Table_aaa26e5e0e2e49a2a6323780ac00c9ea [id=23231f74e45e4d3484db472d381f97f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.518208 28653 ts_tablet_manager.cc:1888] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.615351 28653 log.cc:1196] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/77f0081cc2cb48aaa368741109032fd1
I20240503 14:12:21.525817 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d8f09130c04698949adc45f8b071d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.615909 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 927d4359637f4ec189a5143e8201c7f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.616065 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f9f10348014293a421ed11a50f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.616255 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410e8b6b511403ea609b1353855937b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.616379 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 3112560e16f14b4e89aae90c014be9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.526655 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 005599a6f11843aca07307eaf91e7924 (DEFAULT_TABLE table=default.Table_34131ceab94648048a945b75d3042a55 [id=ccd353dbb8f94ac1af8a109991e98aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.533344 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 31721f852044499b84ad17c4b31964a4 (DEFAULT_TABLE table=default.Table_521b7e40755f4748ad55fc13b690b642 [id=2cf40af77c234dae91434a6034d7d51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.616451 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.533535 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a081ba59ad44890bf2f94874c034277 with delete_type TABLET_DATA_DELETED (replaced by tablet d859044b70fc48e8b62bf6a82ed69d9f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.534479 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.537781 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 95c97a43bd714da68c1a77408f3d1cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.539731 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5e3649c71f4d4da710c69f85a82c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.545689 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dac592c9dd4888a2ce89fa922f7fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.545969 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6c998e518541248a87e4c1551ec080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.616909 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.553050 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet 8097ba5606cf42d68448833de36d85c7 with delete_type TABLET_DATA_DELETED (replaced by tablet c82c2edb1809410c9f11ef639e723db6) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.617386 29848 consensus_queue.cc:1035] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.617543 29867 raft_consensus.cc:2726] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.555581 28600 tablet_bootstrap.cc:492] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:21.577867 28210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.619550 28210 raft_consensus.cc:2454] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:21.578274 28206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf21f6234dd64cf8a32f1599fe50746e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.619941 28206 raft_consensus.cc:2454] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:21.620121 28150 leader_election.cc:341] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.578722 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.620276 28205 raft_consensus.cc:2454] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:21.620587 28150 leader_election.cc:341] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.620955 29848 consensus_queue.cc:1035] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.619959 28600 ts_tablet_manager.cc:1367] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 1.092s user 0.008s sys 0.013s
I20240503 14:12:21.622450 29841 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:21.623860 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5b973ea16d42babd2cc75e32624854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.624171 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad21d284e794156a5b816897d807a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.629645 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 44b8cc9a520a4ab38cd53fb2ab9789b5 (DEFAULT_TABLE table=default.Table_d8070970f59847d091fc2692c79d2ae9 [id=aefbbf7fce604654a8b6f8ca2e872f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.630323 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 7e644199208a45db99c085cd356a0584 (DEFAULT_TABLE table=default.Table_3838859d5d884075bc288faedfd9ce98 [id=a683c6d18deb4968ade55b5edd730f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.615701 28653 ts_tablet_manager.cc:1909] T 77f0081cc2cb48aaa368741109032fd1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:21.631350 28184 tablet_service.cc:1467] Processing CreateTablet for tablet cce4c617ee694c36b3e63d51d6e0f209 (DEFAULT_TABLE table=default.Table_d039e7aa896743fe87bb9365dfe8a704 [id=896cb6b2e8b74e16960f3c6577ed8ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.633011 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.635316 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005599a6f11843aca07307eaf91e7924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.635756 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 429cbd2310c940ad9a8d7a4f4ef17b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.636005 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb6d931156b492e82140ee342cb1bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.636233 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 10876f4512794e6aa72bda3c1da8b2d0
I20240503 14:12:21.636390 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b8cc9a520a4ab38cd53fb2ab9789b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.636809 28031 catalog_manager.cc:6554] ReplaceTablet: tablet 10876f4512794e6aa72bda3c1da8b2d0 deleted and replaced by tablet 5ac145bcc44b4437b840df389cd254f9
W20240503 14:12:21.636847 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 843f0b87687340558d7c6acbbad2c4d7 consensus state has no leader
I20240503 14:12:21.637003 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e644199208a45db99c085cd356a0584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.637312 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 281667db9a9c43d3b9d7dd9ee14b4b06
I20240503 14:12:21.638281 28184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce4c617ee694c36b3e63d51d6e0f209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.638278 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 752b369f24224f2486834814a9d08495
I20240503 14:12:21.638806 28030 catalog_manager.cc:6554] ReplaceTablet: tablet 752b369f24224f2486834814a9d08495 deleted and replaced by tablet ed8cda887ec7497c9950c182e5c2852b
I20240503 14:12:21.638828 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31721f852044499b84ad17c4b31964a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.639032 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da204b27173840f4b0ae09166dabbe13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.639257 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de9edfb0d064939bda2ec7a9e0e500b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.639221 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6874d970e4354070b11060c6b861d7bf
I20240503 14:12:21.640014 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d952b436198141ad80fef5fb1ea1e736
I20240503 14:12:21.641708 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 60d4f5591b1f4826bc09e57094fb46bf
I20240503 14:12:21.642179 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f6d907b2cd874cd98d61854bb4c623fc" } new_table_name: "default.Table_c275691b7f114b36a0915e8338895451" modify_external_catalogs: true
I20240503 14:12:21.643808 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4
I20240503 14:12:21.644433 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d952b436198141ad80fef5fb1ea1e736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.644469 28029 catalog_manager.cc:6554] ReplaceTablet: tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 deleted and replaced by tablet ea5221fc8d6548e08e4ae2c2d466f970
I20240503 14:12:21.646608 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04fff656f624001a709f783816eec1e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:21.659021 29437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.252:45363, user_credentials={real_user=slave}} blocked reactor thread for 45399.8us
I20240503 14:12:21.661321 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 10876f4512794e6aa72bda3c1da8b2d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ac145bcc44b4437b840df389cd254f9) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.661510 29828 ts_tablet_manager.cc:1888] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.664654 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet 10876f4512794e6aa72bda3c1da8b2d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ac145bcc44b4437b840df389cd254f9) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.665877 29828 log.cc:1196] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6c7c5dfac02d4dfe932996d2a37fe031
I20240503 14:12:21.667587 29828 ts_tablet_manager.cc:1909] T 6c7c5dfac02d4dfe932996d2a37fe031 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.667650 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 719) took 3280 ms. Trace:
I20240503 14:12:21.667771 28653 rpcz_store.cc:276] 0503 14:12:18.386738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.387216 (+ 478us) service_pool.cc:225] Handling call
0503 14:12:20.883305 (+2496089us) ts_tablet_manager.cc:1120] Deleting tablet 77f0081cc2cb48aaa368741109032fd1
0503 14:12:20.883318 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.225999 (+342681us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.518065 (+292066us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.667620 (+149555us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11484,"tablet-delete.queue_time_us":2495826}
I20240503 14:12:21.677160 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.678076 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 752b369f24224f2486834814a9d08495 with delete_type TABLET_DATA_DELETED (replaced by tablet ed8cda887ec7497c9950c182e5c2852b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.677966 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_85137044068a45ce8677074d642ad58b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.679311 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ef2d2d784a834891be22d51c1d9f7ae2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.679983 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef2d2d784a834891be22d51c1d9f7ae2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.680279 28653 tablet_replica.cc:330] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:21.680498 28653 raft_consensus.cc:2229] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.680552 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 5d69cc969ea949a684a7e5aae237c13e (table default.Table_b4868a6959c44bfbaad9b187dd171f41 [id=0114799f37994a3396ee46fd1f2e98b5]). Expected version 2 got 0
I20240503 14:12:21.680668 28653 raft_consensus.cc:2258] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.681262 28653 ts_tablet_manager.cc:1875] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.681756 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8bbb6d64decd4c3897319ca3e7130213" } new_table_name: "default.Table_bfff85704d88463383cbf94f9b121359" modify_external_catalogs: true
I20240503 14:12:21.682214 29828 tablet_replica.cc:330] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.682411 29828 raft_consensus.cc:2229] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.682588 29828 raft_consensus.cc:2258] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.683188 29828 ts_tablet_manager.cc:1875] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.683411 29486 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" instance_seqno: 1714745518079174) as {username='slave'} at 127.27.50.193:51061; Asking this server to re-register.
I20240503 14:12:21.684561 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e732ed678eff478eb9c213eb0abc07bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.684765 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a6061a2b91e04219adf9ea0d6a4234e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.685210 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6061a2b91e04219adf9ea0d6a4234e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.685523 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_aa95e49d837a43cf9ed5ce0f5858cc45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.686076 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa95e49d837a43cf9ed5ce0f5858cc45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:21.685211 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e732ed678eff478eb9c213eb0abc07bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.687156 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6874d970e4354070b11060c6b861d7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:21.687455 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85137044068a45ce8677074d642ad58b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.687978 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet d952b436198141ad80fef5fb1ea1e736 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.688341 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a1315a4233714b3f81e5a926c45e14c7" } new_table_name: "default.Table_9f2d3cc9c6724dab9ba35ff24e85c236" modify_external_catalogs: true
I20240503 14:12:21.689293 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d4f5591b1f4826bc09e57094fb46bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.689844 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4d29fd37826542f6aee3139a6338e4ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.690021 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 with delete_type TABLET_DATA_DELETED (replaced by tablet ea5221fc8d6548e08e4ae2c2d466f970) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:21.690394 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d29fd37826542f6aee3139a6338e4ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.695955 29867 raft_consensus.cc:482] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.696149 29867 raft_consensus.cc:504] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.696808 29867 leader_election.cc:288] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:21.700971 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 77f0081cc2cb48aaa368741109032fd1 (table default.Table_730611e6ad48480eb3f4498f3e79725b [id=ddb1f8d9ceae4da98dcb259cde8eac8c]) successfully deleted
I20240503 14:12:21.701941 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.615460 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.702466 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6c7c5dfac02d4dfe932996d2a37fe031 (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
I20240503 14:12:21.702637 28489 raft_consensus.cc:2454] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:21.624959 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.700627 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.703106 28491 raft_consensus.cc:2454] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:21.703548 29845 raft_consensus.cc:2726] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:21.703897 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 104062us
I20240503 14:12:21.704321 29845 consensus_queue.cc:1035] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.710683 28614 tablet_bootstrap.cc:492] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.710923 28614 ts_tablet_manager.cc:1367] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.100s user 0.014s sys 0.021s
I20240503 14:12:21.711829 28614 raft_consensus.cc:348] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.712183 29828 ts_tablet_manager.cc:1888] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.712348 29828 log.cc:1196] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/903f11a608b1483887d9b5234a404470
I20240503 14:12:21.712733 29828 ts_tablet_manager.cc:1909] T 903f11a608b1483887d9b5234a404470 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.713567 29828 tablet_replica.cc:330] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.713841 29828 raft_consensus.cc:2229] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.714110 29828 raft_consensus.cc:2258] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.714217 28150 leader_election.cc:302] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.714597 29811 raft_consensus.cc:2781] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.714682 29811 raft_consensus.cc:482] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.714740 29811 raft_consensus.cc:3037] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.714757 29828 ts_tablet_manager.cc:1875] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.712203 28614 raft_consensus.cc:374] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.716742 28614 raft_consensus.cc:729] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.717075 28614 consensus_queue.cc:260] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.717828 28614 ts_tablet_manager.cc:1398] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:21.719702 28614 tablet_bootstrap.cc:492] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.721870 28614 tablet_bootstrap.cc:654] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:21.722594 28427 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 903f11a608b1483887d9b5234a404470
W20240503 14:12:21.724218 28427 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 903f11a608b1483887d9b5234a404470
I20240503 14:12:21.724514 28427 leader_election.cc:302] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.724810 29845 raft_consensus.cc:2781] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.724916 29845 raft_consensus.cc:482] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.725028 29845 raft_consensus.cc:3037] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.725775 29828 ts_tablet_manager.cc:1888] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.726011 29828 log.cc:1196] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/95d9985eecda4f3194c3251216aa58ee
I20240503 14:12:21.726380 29828 ts_tablet_manager.cc:1909] T 95d9985eecda4f3194c3251216aa58ee P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.727288 29828 tablet_replica.cc:330] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.727535 29828 raft_consensus.cc:2229] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.727758 29828 raft_consensus.cc:2258] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.728017 29845 raft_consensus.cc:504] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.728406 29828 ts_tablet_manager.cc:1875] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.728885 29845 leader_election.cc:288] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.733040 28264 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:21.739830 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 6470ea66cc23472c8893cb652f792783 (DEFAULT_TABLE table=default.Table_0362ba1c8df54ec6a99386f1777d063d [id=3eacbdd68ce24361a171453bf921e779]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.740366 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6470ea66cc23472c8893cb652f792783. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:21.741683 28427 consensus_peers.cc:485] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:21.741858 28427 leader_election.cc:302] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:21.742237 28427 leader_election.cc:341] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.742437 28427 leader_election.cc:302] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.742657 29860 raft_consensus.cc:2781] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.742720 29860 raft_consensus.cc:482] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.742771 29860 raft_consensus.cc:3037] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.746101 29860 raft_consensus.cc:504] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.747031 29860 leader_election.cc:288] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.747278 29860 raft_consensus.cc:2781] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.747411 29860 raft_consensus.cc:482] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.747531 29860 raft_consensus.cc:3037] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.750547 29860 raft_consensus.cc:504] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.751402 29860 leader_election.cc:288] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.754067 28323 tablet_service.cc:1467] Processing CreateTablet for tablet d110c15abf1f4770b369c055a934596b (DEFAULT_TABLE table=default.Table_2404de92682c45eda574d316d0440be6 [id=b26871a2d1e54fd89d04e962f16aba78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.754710 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d110c15abf1f4770b369c055a934596b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.759435 28604 tablet_bootstrap.cc:654] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.762015 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 65080b4da42743fba08cd752e7c6d3b9 (DEFAULT_TABLE table=default.Table_41af65ba4c3b41f5924502c36a902d9b [id=abb2727168674e4a9f1466b466517123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.762244 28427 leader_election.cc:302] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.762606 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65080b4da42743fba08cd752e7c6d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.763464 29828 ts_tablet_manager.cc:1888] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.763494 29860 raft_consensus.cc:2781] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:21.764022 29860 raft_consensus.cc:686] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:21.764428 29860 consensus_queue.cc:237] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.767673 29860 raft_consensus.cc:482] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.767916 29860 raft_consensus.cc:504] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.768466 29828 log.cc:1196] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/95c97a43bd714da68c1a77408f3d1cd1
I20240503 14:12:21.768702 29860 leader_election.cc:288] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.777215 28150 leader_election.cc:302] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.777271 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 903f11a608b1483887d9b5234a404470 (table default.Table_4afe0b312f964d3885779a752ec8c256 [id=32be054076954aed9265888ca62e510b]) successfully deleted
I20240503 14:12:21.777571 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 95d9985eecda4f3194c3251216aa58ee (table default.Table_627b41e76476426da74d6722301bc59f [id=97e7cb47edc44a5e9041be7ad4bc6f47]) successfully deleted
W20240503 14:12:21.777863 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 3112560e16f14b4e89aae90c014be9fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3112560e16f14b4e89aae90c014be9fb already in progress: creating tablet
I20240503 14:12:21.778223 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad397ed76ea4cf8820312ba1362512f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.781884 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90e9563b4c4cef8ee7964eaa96e285" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.782177 28205 raft_consensus.cc:2379] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:21.785795 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 752b369f24224f2486834814a9d08495 with delete_type TABLET_DATA_DELETED (replaced by tablet ed8cda887ec7497c9950c182e5c2852b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.786379 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.787556 29750 consensus_queue.cc:1035] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:21.794194 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4654e5b18e14aab9e976088a0939680 consensus state has no leader
I20240503 14:12:21.794486 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 535ddec4ce9f409aaccd6b244deb1596 (DEFAULT_TABLE table=default.Table_ea32369d7dcc4ed5a82c2910a55b4583 [id=59cf84748b1342159832f667892ad76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.794580 28427 leader_election.cc:302] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:21.794999 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535ddec4ce9f409aaccd6b244deb1596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.795480 29701 raft_consensus.cc:2781] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.795567 29701 raft_consensus.cc:482] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.795624 29701 raft_consensus.cc:3037] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.796360 28427 leader_election.cc:302] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:21.801133 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4c7cc36935e640d0bff537b79d5b09d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.802006 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c7cc36935e640d0bff537b79d5b09d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.805049 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet dda39f0323ec47799d3d44c96f3ba605 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:21.806735 28026 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet ea5221fc8d6548e08e4ae2c2d466f970
I20240503 14:12:21.807220 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b1ae9903413045af9a9fa56d22ac1552" } new_table_name: "default.Table_a42d70182460481cbcf7f90a0dc57481" modify_external_catalogs: true
W20240503 14:12:21.811300 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8acbc154c03c47c1b2f9f433fcb2ef9a consensus state has no leader
I20240503 14:12:21.815271 28032 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 5d69cc969ea949a684a7e5aae237c13e (table default.Table_b4868a6959c44bfbaad9b187dd171f41 [id=0114799f37994a3396ee46fd1f2e98b5]). Expected version 2 got 0
I20240503 14:12:21.817523 29701 raft_consensus.cc:504] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.817991 29828 ts_tablet_manager.cc:1909] T 95c97a43bd714da68c1a77408f3d1cd1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:21.818531 29701 leader_election.cc:288] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.819016 28427 leader_election.cc:302] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:21.819216 29901 raft_consensus.cc:2781] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.819582 29750 raft_consensus.cc:2781] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.819655 29750 raft_consensus.cc:482] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.819630 29901 raft_consensus.cc:482] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.819717 29750 raft_consensus.cc:3037] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.819733 29901 raft_consensus.cc:3037] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.821430 28661 ts_tablet_manager.cc:1888] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:21.821563 28661 log.cc:1196] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/00b52ea336524beea2f255c8720f7472
I20240503 14:12:21.821946 28661 ts_tablet_manager.cc:1909] T 00b52ea336524beea2f255c8720f7472 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:21.822062 29701 consensus_queue.cc:1035] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.822970 29750 raft_consensus.cc:504] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.823665 29750 leader_election.cc:288] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:21.819592 28427 leader_election.cc:341] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.824060 28427 leader_election.cc:302] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.827021 28427 leader_election.cc:302] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:21.827494 29860 raft_consensus.cc:2726] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.827669 29860 raft_consensus.cc:2726] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240503 14:12:21.827968 28427 leader_election.cc:341] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:21.828320 28427 leader_election.cc:341] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:21.828600 28427 leader_election.cc:341] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.823957 29901 raft_consensus.cc:504] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.831133 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 4764ce33d6654fa7a6edf9273803ea4a (DEFAULT_TABLE table=default.Table_7b5c8fe5421e45faa3a85b72495c0cae [id=280e8e2e85394dec81b1443a2b9eaf0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.831398 29901 leader_election.cc:288] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.831574 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764ce33d6654fa7a6edf9273803ea4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.824453 29750 raft_consensus.cc:482] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3a6b9b7913444d29c96581af6febf33)
I20240503 14:12:21.775600 29811 raft_consensus.cc:504] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.831673 29750 raft_consensus.cc:504] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:21.832310 29750 leader_election.cc:288] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.834371 28653 ts_tablet_manager.cc:1888] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:21.834504 28653 log.cc:1196] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/bf21f6234dd64cf8a32f1599fe50746e
I20240503 14:12:21.834818 28653 ts_tablet_manager.cc:1909] T bf21f6234dd64cf8a32f1599fe50746e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:21.835520 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 7e644199208a45db99c085cd356a0584 (DEFAULT_TABLE table=default.Table_3838859d5d884075bc288faedfd9ce98 [id=a683c6d18deb4968ade55b5edd730f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.835947 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e644199208a45db99c085cd356a0584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.836104 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b25f8a0acb9f425aa6019f374e8bf5cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:21.836757 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b25f8a0acb9f425aa6019f374e8bf5cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.837082 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_60e99647c73642ceb431834009fa427e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.837244 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 728ba3ac1cb645168481b84b5b548140 (DEFAULT_TABLE table=default.Table_3f8dd5c5498a4ffda480e84d0526e3f4 [id=d238c2b32772436494262ebce1dec72e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:21.837599 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60e99647c73642ceb431834009fa427e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.837625 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728ba3ac1cb645168481b84b5b548140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.838660 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet 86804987bdf340dc83ddd58ba2bc8a96 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:21.839545 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86d810ab0cf34941a343eef1593632bf consensus state has no leader
I20240503 14:12:21.840406 28456 tablet_service.cc:1467] Processing CreateTablet for tablet cce4c617ee694c36b3e63d51d6e0f209 (DEFAULT_TABLE table=default.Table_d039e7aa896743fe87bb9365dfe8a704 [id=896cb6b2e8b74e16960f3c6577ed8ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.840502 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b71e58c4b31248888eecec2d163d0181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:21.840782 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce4c617ee694c36b3e63d51d6e0f209. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:21.841130 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b71e58c4b31248888eecec2d163d0181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:21.843662 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a07e2f0650084270860bd28e40f224aa" } modify_external_catalogs: true
I20240503 14:12:21.843956 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a07e2f0650084270860bd28e40f224aa" } modify_external_catalogs: true
I20240503 14:12:21.844596 28458 tablet_service.cc:1514] Processing DeleteTablet for tablet 6874d970e4354070b11060c6b861d7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.845093 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 535ddec4ce9f409aaccd6b244deb1596 (DEFAULT_TABLE table=default.Table_ea32369d7dcc4ed5a82c2910a55b4583 [id=59cf84748b1342159832f667892ad76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.845491 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535ddec4ce9f409aaccd6b244deb1596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.845537 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5a4ea581b23745598fabca5d5e715f73" } modify_external_catalogs: true
I20240503 14:12:21.845736 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5a4ea581b23745598fabca5d5e715f73" } modify_external_catalogs: true
I20240503 14:12:21.846323 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet dda39f0323ec47799d3d44c96f3ba605
I20240503 14:12:21.846498 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 65080b4da42743fba08cd752e7c6d3b9 (DEFAULT_TABLE table=default.Table_41af65ba4c3b41f5924502c36a902d9b [id=abb2727168674e4a9f1466b466517123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.846843 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65080b4da42743fba08cd752e7c6d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.846918 28027 catalog_manager.cc:6554] ReplaceTablet: tablet dda39f0323ec47799d3d44c96f3ba605 deleted and replaced by tablet e750765798db487fb0b5142f01d21771
I20240503 14:12:21.847051 29486 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" instance_seqno: 1714745519449060) as {username='slave'} at 127.27.50.195:34397; Asking this server to re-register.
I20240503 14:12:21.847275 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8e4c3260d34f48dba9b1270f6391cb17" } new_table_name: "default.Table_a7060c6b06e14403ad4ab81fc6e5c8cd" modify_external_catalogs: true
I20240503 14:12:21.847574 28462 tablet_service.cc:1467] Processing CreateTablet for tablet d110c15abf1f4770b369c055a934596b (DEFAULT_TABLE table=default.Table_2404de92682c45eda574d316d0440be6 [id=b26871a2d1e54fd89d04e962f16aba78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.849649 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 752b369f24224f2486834814a9d08495 with delete_type TABLET_DATA_DELETED (replaced by tablet ed8cda887ec7497c9950c182e5c2852b) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.849714 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_34e94ee61c2644d3b8f4449f0d6f341b" } modify_external_catalogs: true
I20240503 14:12:21.844300 28474 tablet_service.cc:1514] Processing DeleteTablet for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.849916 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_34e94ee61c2644d3b8f4449f0d6f341b" } modify_external_catalogs: true
I20240503 14:12:21.854300 29867 raft_consensus.cc:482] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.854481 29867 raft_consensus.cc:504] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.855093 29867 leader_election.cc:288] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.856448 29811 leader_election.cc:288] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:21.856539 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 5d69cc969ea949a684a7e5aae237c13e (table default.Table_b4868a6959c44bfbaad9b187dd171f41 [id=0114799f37994a3396ee46fd1f2e98b5]). Expected version 2 got 0
I20240503 14:12:21.856639 29811 raft_consensus.cc:2781] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.856709 29811 raft_consensus.cc:482] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.856766 29811 raft_consensus.cc:3037] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.856797 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet e6c90869b5034128864faaf05697c230 (table default.Table_5dcdff74f1f7407084a45dc9fe588103 [id=47ade8624b00424d8fc04dbca5174ff4]). Expected version 3 got 0
I20240503 14:12:21.857884 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9aebe1270f6749608f4ec392cecede93" } modify_external_catalogs: true
I20240503 14:12:21.861414 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9aebe1270f6749608f4ec392cecede93" } modify_external_catalogs: true
I20240503 14:12:21.866376 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet d952b436198141ad80fef5fb1ea1e736 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:21.867323 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff07bd9bd59f4b13955904cc172c044c consensus state has no leader
I20240503 14:12:21.867908 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d4f5591b1f4826bc09e57094fb46bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.870378 29811 raft_consensus.cc:504] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.814134 28290 leader_election.cc:302] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:21.873634 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 4764ce33d6654fa7a6edf9273803ea4a (DEFAULT_TABLE table=default.Table_7b5c8fe5421e45faa3a85b72495c0cae [id=280e8e2e85394dec81b1443a2b9eaf0c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:21.873858 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 59799us
I20240503 14:12:21.873924 29867 raft_consensus.cc:3037] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.875134 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 728ba3ac1cb645168481b84b5b548140 (DEFAULT_TABLE table=default.Table_3f8dd5c5498a4ffda480e84d0526e3f4 [id=d238c2b32772436494262ebce1dec72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.875466 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet dda39f0323ec47799d3d44c96f3ba605 with delete_type TABLET_DATA_DELETED (replaced by tablet e750765798db487fb0b5142f01d21771) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.879488 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5297664a4d943c28adfb6ac4f061680" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:21.881098 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 44b8cc9a520a4ab38cd53fb2ab9789b5 (DEFAULT_TABLE table=default.Table_d8070970f59847d091fc2692c79d2ae9 [id=aefbbf7fce604654a8b6f8ca2e872f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.883698 29867 raft_consensus.cc:2726] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.884681 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cdd090bf0f493f8b92341369378532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.872954 28184 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 with delete_type TABLET_DATA_DELETED (replaced by tablet ea5221fc8d6548e08e4ae2c2d466f970) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.885052 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b8cc9a520a4ab38cd53fb2ab9789b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.871302 28600 raft_consensus.cc:348] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.885203 28600 raft_consensus.cc:374] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.885290 28600 raft_consensus.cc:729] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.885504 28600 consensus_queue.cc:260] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.886119 28600 ts_tablet_manager.cc:1398] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.264s user 0.001s sys 0.001s
I20240503 14:12:21.875147 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 535ddec4ce9f409aaccd6b244deb1596 (DEFAULT_TABLE table=default.Table_ea32369d7dcc4ed5a82c2910a55b4583 [id=59cf84748b1342159832f667892ad76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.875550 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 6470ea66cc23472c8893cb652f792783 (DEFAULT_TABLE table=default.Table_0362ba1c8df54ec6a99386f1777d063d [id=3eacbdd68ce24361a171453bf921e779]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.875873 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 65080b4da42743fba08cd752e7c6d3b9 (DEFAULT_TABLE table=default.Table_41af65ba4c3b41f5924502c36a902d9b [id=abb2727168674e4a9f1466b466517123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.876188 28192 tablet_service.cc:1467] Processing CreateTablet for tablet d110c15abf1f4770b369c055a934596b (DEFAULT_TABLE table=default.Table_2404de92682c45eda574d316d0440be6 [id=b26871a2d1e54fd89d04e962f16aba78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.876477 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 4764ce33d6654fa7a6edf9273803ea4a (DEFAULT_TABLE table=default.Table_7b5c8fe5421e45faa3a85b72495c0cae [id=280e8e2e85394dec81b1443a2b9eaf0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.876752 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 728ba3ac1cb645168481b84b5b548140 (DEFAULT_TABLE table=default.Table_3f8dd5c5498a4ffda480e84d0526e3f4 [id=d238c2b32772436494262ebce1dec72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.876996 28194 tablet_service.cc:1514] Processing DeleteTablet for tablet dda39f0323ec47799d3d44c96f3ba605 with delete_type TABLET_DATA_DELETED (replaced by tablet e750765798db487fb0b5142f01d21771) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.878438 29892 tablet.cc:1722] T f2b861dccd964092ae0f06668c9b58fe P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:21.887439 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 with delete_type TABLET_DATA_DELETED (replaced by tablet ea5221fc8d6548e08e4ae2c2d466f970) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.822883 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 736) took 3247 ms. Trace:
I20240503 14:12:21.887660 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d4f5591b1f4826bc09e57094fb46bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.888542 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d110c15abf1f4770b369c055a934596b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.887815 28661 rpcz_store.cc:276] 0503 14:12:18.575087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.575146 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:21.396382 (+2821236us) ts_tablet_manager.cc:1120] Deleting tablet 00b52ea336524beea2f255c8720f7472
0503 14:12:21.396391 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.597734 (+201343us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.821272 (+223538us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.822849 (+ 1577us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3350,"tablet-delete.queue_time_us":2816663}
I20240503 14:12:21.883044 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet 6874d970e4354070b11060c6b861d7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.891045 29811 leader_election.cc:288] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:21.892410 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 41cbd1326869434594e126421ad132d8
W20240503 14:12:21.894500 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e220336186f648419b186feada6f0124 consensus state has no leader
I20240503 14:12:21.897009 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b04fff656f624001a709f783816eec1e
I20240503 14:12:21.887754 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet d952b436198141ad80fef5fb1ea1e736 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.904048 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a081ba59ad44890bf2f94874c034277" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.903344 28661 tablet_replica.cc:330] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:21.905026 28488 raft_consensus.cc:2454] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:21.904186 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 10876f4512794e6aa72bda3c1da8b2d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ac145bcc44b4437b840df389cd254f9) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:21.904515 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 6470ea66cc23472c8893cb652f792783 (DEFAULT_TABLE table=default.Table_0362ba1c8df54ec6a99386f1777d063d [id=3eacbdd68ce24361a171453bf921e779]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.906349 28290 leader_election.cc:302] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:21.906603 28661 raft_consensus.cc:2229] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:21.909669 28661 pending_rounds.cc:62] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:21.910296 28661 pending_rounds.cc:68] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597723663011840 op_type: NO_OP noop_request { }
I20240503 14:12:21.911545 28661 raft_consensus.cc:2866] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:21.912957 29867 raft_consensus.cc:2781] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.913182 29867 raft_consensus.cc:482] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.912954 28661 raft_consensus.cc:2258] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.913317 29867 raft_consensus.cc:3037] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.916894 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764ce33d6654fa7a6edf9273803ea4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.917796 29828 tablet_replica.cc:330] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:21.919371 29892 tablet.cc:1722] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:21.920914 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728ba3ac1cb645168481b84b5b548140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.925683 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a081ba59ad44890bf2f94874c034277" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.925958 28205 raft_consensus.cc:2454] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:21.929530 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728ba3ac1cb645168481b84b5b548140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.916021 29867 raft_consensus.cc:504] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.931856 29867 leader_election.cc:288] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:21.931988 29828 raft_consensus.cc:2229] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.932374 29828 raft_consensus.cc:2258] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:21.933470 29828 ts_tablet_manager.cc:1875] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.934146 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6470ea66cc23472c8893cb652f792783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.935092 28206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a081ba59ad44890bf2f94874c034277" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:21.935318 28206 raft_consensus.cc:3037] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.935748 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet b04fff656f624001a709f783816eec1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.936000 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 41cbd1326869434594e126421ad132d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:21.938196 28206 raft_consensus.cc:2454] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:21.938792 28290 leader_election.cc:302] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
W20240503 14:12:21.939007 28290 raft_consensus.cc:2670] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:21.940721 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 717) took 3554 ms. Trace:
I20240503 14:12:21.940032 28661 ts_tablet_manager.cc:1875] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.940819 28653 rpcz_store.cc:276] 0503 14:12:18.386442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.412131 (+ 25689us) service_pool.cc:225] Handling call
0503 14:12:21.680213 (+3268082us) ts_tablet_manager.cc:1120] Deleting tablet bf21f6234dd64cf8a32f1599fe50746e
0503 14:12:21.680226 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.699210 (+ 18984us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.834235 (+135025us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:21.940680 (+106445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3265745}
I20240503 14:12:21.941263 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 727) took 3299 ms. Trace:
I20240503 14:12:21.941341 28653 rpcz_store.cc:276] 0503 14:12:18.641719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.641780 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:21.941101 (+3299321us) ts_tablet_manager.cc:1120] Deleting tablet f67c4cb980b44e9397bea70363fe1acd
0503 14:12:21.941112 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.941243 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258241}
I20240503 14:12:21.941692 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 725) took 3330 ms. Trace:
I20240503 14:12:21.941764 28653 rpcz_store.cc:276] 0503 14:12:18.610803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.610878 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:21.941583 (+3330705us) ts_tablet_manager.cc:1120] Deleting tablet 2d5192083ed94a3d8d2d1a0bf001a1eb
0503 14:12:21.941593 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.941676 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258605}
I20240503 14:12:21.942044 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 726) took 3306 ms. Trace:
I20240503 14:12:21.942103 28653 rpcz_store.cc:276] 0503 14:12:18.635717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.637600 (+ 1883us) service_pool.cc:225] Handling call
0503 14:12:21.941951 (+3304351us) ts_tablet_manager.cc:1120] Deleting tablet a6b468dd4102495abd4337edcfc6340a
0503 14:12:21.941959 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.942031 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258882}
I20240503 14:12:21.942373 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 728) took 3273 ms. Trace:
I20240503 14:12:21.942435 28653 rpcz_store.cc:276] 0503 14:12:18.668979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.670293 (+ 1314us) service_pool.cc:225] Handling call
0503 14:12:21.942279 (+3271986us) ts_tablet_manager.cc:1120] Deleting tablet 0a6a58db50f849a8be729c10dbe03e76
0503 14:12:21.942286 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.942357 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3255610}
I20240503 14:12:21.942718 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 730) took 3159 ms. Trace:
I20240503 14:12:21.941267 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 95c97a43bd714da68c1a77408f3d1cd1 (table default.Table_c7035b3f783948498dfb7bd5ed14aad8 [id=15ae3079abdd4069aaeb84e1d85da988]) successfully deleted
I20240503 14:12:21.942786 28653 rpcz_store.cc:276] 0503 14:12:18.783324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.783609 (+ 285us) service_pool.cc:225] Handling call
0503 14:12:21.942617 (+3159008us) ts_tablet_manager.cc:1120] Deleting tablet ac577af6460b423288a790e532b00f82
0503 14:12:21.942624 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.942693 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156022}
I20240503 14:12:21.943063 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 729) took 3209 ms. Trace:
I20240503 14:12:21.943122 28653 rpcz_store.cc:276] 0503 14:12:18.783145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.784141 (+ 996us) service_pool.cc:225] Handling call
0503 14:12:21.942968 (+3158827us) ts_tablet_manager.cc:1120] Deleting tablet 7927f3e0fb214441aaf52352b6956ef6
0503 14:12:21.942977 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.943047 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3155582}
I20240503 14:12:21.933720 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a081ba59ad44890bf2f94874c034277" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:21.943393 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 731) took 3156 ms. Trace:
I20240503 14:12:21.943414 28488 raft_consensus.cc:3037] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.943457 28653 rpcz_store.cc:276] 0503 14:12:18.786840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.786997 (+ 157us) service_pool.cc:225] Handling call
0503 14:12:21.943302 (+3156305us) ts_tablet_manager.cc:1120] Deleting tablet e69e01bdbae349d0a61a295b6f6352f0
0503 14:12:21.943310 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:21.943379 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3148181}
I20240503 14:12:21.943739 28653 tablet_replica.cc:330] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:21.943935 28653 raft_consensus.cc:2229] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:21.944448 28614 tablet_bootstrap.cc:492] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:21.945099 28600 tablet_bootstrap.cc:492] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:21.945809 28614 ts_tablet_manager.cc:1367] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.226s user 0.053s sys 0.093s
I20240503 14:12:21.946380 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet 41cbd1326869434594e126421ad132d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.946808 28178 tablet_service.cc:1514] Processing DeleteTablet for tablet b04fff656f624001a709f783816eec1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:21.946818 28600 tablet_bootstrap.cc:654] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.949491 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764ce33d6654fa7a6edf9273803ea4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.952430 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d110c15abf1f4770b369c055a934596b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.952704 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535ddec4ce9f409aaccd6b244deb1596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.956310 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65080b4da42743fba08cd752e7c6d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.957397 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a92fc3833e49aaab8c5415de9bb685" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:21.957615 28211 raft_consensus.cc:3037] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.957151 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet bf21f6234dd64cf8a32f1599fe50746e (table default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]) successfully deleted
I20240503 14:12:21.956352 28614 raft_consensus.cc:348] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.958878 28614 raft_consensus.cc:374] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.959471 28614 raft_consensus.cc:729] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.960220 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6470ea66cc23472c8893cb652f792783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.960691 28211 raft_consensus.cc:2454] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:21.961230 29867 raft_consensus.cc:482] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:21.960703 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f67c4cb980b44e9397bea70363fe1acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f67c4cb980b44e9397bea70363fe1acd already in progress: creating tablet
I20240503 14:12:21.961356 29867 raft_consensus.cc:504] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.962536 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af0342e0e0b47499a9322b5b2cb3d03" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:21.961687 28614 consensus_queue.cc:260] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:21.963246 28290 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.964917 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495190fca06a40a386c8215eb2195920" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:21.966380 28653 raft_consensus.cc:2258] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:21.965780 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d5192083ed94a3d8d2d1a0bf001a1eb already in progress: creating tablet
I20240503 14:12:21.967841 28653 ts_tablet_manager.cc:1875] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:21.970129 28614 ts_tablet_manager.cc:1398] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.024s user 0.002s sys 0.002s
I20240503 14:12:21.972761 28600 tablet_bootstrap.cc:492] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:21.972544 28614 tablet_bootstrap.cc:492] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:21.973054 28600 ts_tablet_manager.cc:1367] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.008s
I20240503 14:12:21.973809 28600 raft_consensus.cc:348] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.974104 28600 raft_consensus.cc:374] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.974217 28600 raft_consensus.cc:729] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.974506 28600 consensus_queue.cc:260] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.975391 28600 ts_tablet_manager.cc:1398] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 14:12:21.969977 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a6b468dd4102495abd4337edcfc6340a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b468dd4102495abd4337edcfc6340a already in progress: creating tablet
I20240503 14:12:21.976788 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bae128469f4e8bb2d743320731446a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:21.976348 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0a6a58db50f849a8be729c10dbe03e76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6a58db50f849a8be729c10dbe03e76 already in progress: creating tablet
I20240503 14:12:21.977169 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:21.978147 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ac577af6460b423288a790e532b00f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac577af6460b423288a790e532b00f82 already in progress: creating tablet
I20240503 14:12:21.980311 28206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495190fca06a40a386c8215eb2195920" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:21.980532 28206 raft_consensus.cc:3037] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:21.980044 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7927f3e0fb214441aaf52352b6956ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7927f3e0fb214441aaf52352b6956ef6 already in progress: creating tablet
W20240503 14:12:21.982290 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e69e01bdbae349d0a61a295b6f6352f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e69e01bdbae349d0a61a295b6f6352f0 already in progress: creating tablet
I20240503 14:12:21.984532 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:21.984889 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf21f6234dd64cf8a32f1599fe50746e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:21.985515 28210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bae128469f4e8bb2d743320731446a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.985738 28210 raft_consensus.cc:2454] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:21.987422 28600 tablet_bootstrap.cc:492] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:21.989470 28600 tablet_bootstrap.cc:654] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.990628 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:21.990765 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:21.990887 28217 raft_consensus.cc:2379] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:21.990980 28216 raft_consensus.cc:2379] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:21.991910 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9011de10a7ff497fb35ccb3ada7315ef" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:21.992424 28208 raft_consensus.cc:2454] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:21.997781 28600 tablet_bootstrap.cc:492] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:21.998064 28600 ts_tablet_manager.cc:1367] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240503 14:12:21.998958 28600 raft_consensus.cc:348] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:21.999292 28600 raft_consensus.cc:374] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.999389 28600 raft_consensus.cc:729] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.999691 28600 consensus_queue.cc:260] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.000305 28600 ts_tablet_manager.cc:1398] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:22.001339 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f89fca0480d4bf28160384ae0421bce" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:22.001595 28211 raft_consensus.cc:3037] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.001534 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b07a739f381c48e28998d3e15695f45f" } new_table_name: "default.Table_d47ff25520da4d3c87c340cdaae46100" modify_external_catalogs: true
I20240503 14:12:22.004586 28600 tablet_bootstrap.cc:492] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.006340 28600 tablet_bootstrap.cc:654] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.006279 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_865c53e9072b4f2d81c2809a817e35fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.007073 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_865c53e9072b4f2d81c2809a817e35fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:22.009733 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 137c9eef465843049e93374367e562af consensus state has no leader
I20240503 14:12:22.015558 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_94d6ea5674494c8a960eaee9c10ded44" } modify_external_catalogs: true
I20240503 14:12:22.015847 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_94d6ea5674494c8a960eaee9c10ded44" } modify_external_catalogs: true
I20240503 14:12:22.017634 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0ddd72e685204e87a5edb2761dda0b0c" } new_table_name: "default.Table_de7c489221b549229264191fa22fffac" modify_external_catalogs: true
I20240503 14:12:22.019465 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e141d550a34a4f838e7db3335d0ea5ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.024561 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e141d550a34a4f838e7db3335d0ea5ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.031986 28033 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet c82c2edb1809410c9f11ef639e723db6
W20240503 14:12:22.033231 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d4c5648ddc84407a8d8b44f2d63dcee consensus state has no leader
I20240503 14:12:22.036664 28206 raft_consensus.cc:2454] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:22.038563 29867 leader_election.cc:288] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.043102 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_85a520f4fd804e969bbb4149159976e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:22.040175 28488 raft_consensus.cc:2454] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
W20240503 14:12:22.043879 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85a520f4fd804e969bbb4149159976e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.045331 28604 tablet_bootstrap.cc:492] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:22.045493 28604 ts_tablet_manager.cc:1367] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.466s user 0.018s sys 0.007s
I20240503 14:12:22.046909 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3ede1f6683664f5d9956c6f825d3d7c8" } new_table_name: "default.Table_35e27f7a242d4ffaa19a6e07ec525559" modify_external_catalogs: true
I20240503 14:12:22.049206 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2841ebe2e868418690799c3ccd7901df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.050289 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2841ebe2e868418690799c3ccd7901df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.050524 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6650c1de6ff4404ca4f4a85acd9b2204"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.051520 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6650c1de6ff4404ca4f4a85acd9b2204 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.053197 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d6956bfb321b412dac6347c07258731c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.054131 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6956bfb321b412dac6347c07258731c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.061141 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_06fe0125c0644021bf434b53126d7f3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.061897 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06fe0125c0644021bf434b53126d7f3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.062754 28604 raft_consensus.cc:348] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.063021 28604 raft_consensus.cc:374] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.063074 28604 raft_consensus.cc:729] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.063310 28604 consensus_queue.cc:260] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.063899 28604 ts_tablet_manager.cc:1398] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 14:12:22.064294 28604 tablet_bootstrap.cc:492] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:22.065678 28211 raft_consensus.cc:2454] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:22.067575 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 352362us
I20240503 14:12:22.069471 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:22.069514 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:22.069782 28493 raft_consensus.cc:2379] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:22.070744 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_425624bfb64a4882893b569d3044c2c0" } new_table_name: "default.Table_7b05c9498d684b81b922fec65df03995" modify_external_catalogs: true
I20240503 14:12:22.071126 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:22.071405 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:22.071614 28488 raft_consensus.cc:2379] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
W20240503 14:12:22.072309 28290 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:22.072604 28290 leader_election.cc:302] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:22.073267 29864 raft_consensus.cc:2726] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.074689 29750 raft_consensus.cc:482] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.074864 29750 raft_consensus.cc:504] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:22.075637 29750 leader_election.cc:288] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.077746 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 00b52ea336524beea2f255c8720f7472 (table default.Table_9bcbc7e8d1154ffd94951feb50445bc2 [id=eb3d80381dc844dcac4ed2bc171565d0]) successfully deleted
I20240503 14:12:22.082576 28357 raft_consensus.cc:3037] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.084113 29486 ts_manager.cc:194] Registered new tserver with Master: a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.088297 29892 tablet.cc:1722] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240503 14:12:22.092424 28150 leader_election.cc:341] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.095372 29915 tablet.cc:1722] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:22.096285 28150 leader_election.cc:302] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:22.097944 29811 raft_consensus.cc:2726] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240503 14:12:22.098976 28150 leader_election.cc:341] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.101708 28150 leader_election.cc:302] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:22.102784 29811 raft_consensus.cc:2726] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.105572 29913 tablet.cc:1722] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:22.107362 28614 tablet_bootstrap.cc:654] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.108711 29892 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58172 (request call id 816) took 1018 ms. Trace:
I20240503 14:12:22.111770 29828 ts_tablet_manager.cc:1888] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.111961 29828 log.cc:1196] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/5a081ba59ad44890bf2f94874c034277
I20240503 14:12:22.102196 28604 tablet_bootstrap.cc:654] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.112682 29892 rpcz_store.cc:276] 0503 14:12:21.090097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.206242 (+116145us) service_pool.cc:225] Handling call
0503 14:12:22.108675 (+902433us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:21.444358 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:21.444413 (+ 55us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:21.444419 (+ 6us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:21.444423 (+ 4us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:21.444462 (+ 39us) alter_schema_op.cc:119] START. Timestamp: 1714745541444433
0503 14:12:21.444466 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:21.444742 (+ 276us) log.cc:842] Serialized 173 byte log entry
0503 14:12:22.087794 (+643052us) op_driver.cc:444] REPLICATION: finished
0503 14:12:22.088261 (+ 467us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:22.090847 (+ 2586us) log.cc:842] Serialized 5 byte log entry
0503 14:12:22.095894 (+ 5047us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.108372 (+ 12478us) log.cc:842] Serialized 14 byte log entry
0503 14:12:22.108634 (+ 262us) alter_schema_op.cc:65] Released schema lock
0503 14:12:22.108638 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":391,"prepare.queue_time_us":237738,"prepare.run_cpu_time_us":5130,"prepare.run_wall_time_us":155961,"raft.queue_time_us":52,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":643290,"thread_start_us":429708,"threads_started":2,"wal-append.queue_time_us":147682}]]}
I20240503 14:12:22.117511 28604 tablet_bootstrap.cc:492] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:22.117807 28604 ts_tablet_manager.cc:1367] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.001s
I20240503 14:12:22.118985 28604 raft_consensus.cc:348] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.119855 28604 raft_consensus.cc:374] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.120043 28604 raft_consensus.cc:729] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.121582 28604 consensus_queue.cc:260] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.122509 28604 ts_tablet_manager.cc:1398] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:22.122924 28604 tablet_bootstrap.cc:492] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:22.125351 28604 tablet_bootstrap.cc:654] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.132532 29913 tablet.cc:1722] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:22.137842 28600 tablet_bootstrap.cc:492] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:22.138123 28600 ts_tablet_manager.cc:1367] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.134s user 0.011s sys 0.016s
I20240503 14:12:22.139012 28600 raft_consensus.cc:348] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.139319 28600 raft_consensus.cc:374] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.139432 28600 raft_consensus.cc:729] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
W20240503 14:12:22.143122 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 725b4c9a47c0458ba188c0cf62f3fe48 consensus state has no leader
I20240503 14:12:22.141285 29892 tablet.cc:1717] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:22.147403 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_715482091e944a8abe222f740b5be62b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:22.147284 29892 tablet.cc:1722] T 5d69cc969ea949a684a7e5aae237c13e P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:22.148171 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_715482091e944a8abe222f740b5be62b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.151626 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_30c6122e36244089a4eba499c8881c3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.154595 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30c6122e36244089a4eba499c8881c3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.157140 28614 tablet_bootstrap.cc:492] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.157346 28614 ts_tablet_manager.cc:1367] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.185s user 0.002s sys 0.016s
I20240503 14:12:22.158337 28614 raft_consensus.cc:348] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.158638 28614 raft_consensus.cc:374] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.158700 28614 raft_consensus.cc:729] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.159108 29913 tablet.cc:1722] T 5d69cc969ea949a684a7e5aae237c13e P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:22.141326 28653 ts_tablet_manager.cc:1888] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.160246 28653 log.cc:1196] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/5dd8b2adaeeb4c3abfbfe68cc005e03a
I20240503 14:12:22.142911 28600 consensus_queue.cc:260] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.160616 28653 ts_tablet_manager.cc:1909] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.160907 28600 ts_tablet_manager.cc:1398] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240503 14:12:22.161319 28600 tablet_bootstrap.cc:492] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.161518 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 740) took 2909 ms. Trace:
I20240503 14:12:22.161628 28653 rpcz_store.cc:276] 0503 14:12:19.252499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.252574 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:21.943684 (+2691110us) ts_tablet_manager.cc:1120] Deleting tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a
0503 14:12:21.943693 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.010941 (+ 67248us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.141209 (+130268us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.161492 (+ 20283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2690809}
I20240503 14:12:22.162261 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a (table default.Table_79df8c9603ae456799a73fb5badd18e9 [id=23dabedfe8df4e9fa6123f2dd9b82376]) successfully deleted
I20240503 14:12:22.158934 28614 consensus_queue.cc:260] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.163059 28614 ts_tablet_manager.cc:1398] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:12:22.163493 28614 tablet_bootstrap.cc:492] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.168403 29919 tablet.cc:1717] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 3, tablet already version 3
W20240503 14:12:22.079430 28427 leader_election.cc:341] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: c5297664a4d943c28adfb6ac4f061680
W20240503 14:12:22.171288 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 91891.7us
I20240503 14:12:22.171756 28427 leader_election.cc:302] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:22.172286 29901 raft_consensus.cc:2781] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.172405 29901 raft_consensus.cc:686] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:22.173714 29919 tablet.cc:1717] T e6c90869b5034128864faaf05697c230 P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:22.173733 28427 leader_election.cc:302] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:22.174178 29750 raft_consensus.cc:2781] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.174263 29750 raft_consensus.cc:482] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.174319 29750 raft_consensus.cc:3037] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.175577 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 745) took 2920 ms. Trace:
I20240503 14:12:22.175760 28653 rpcz_store.cc:276] 0503 14:12:19.255095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.255207 (+ 112us) service_pool.cc:225] Handling call
0503 14:12:22.175399 (+2920192us) ts_tablet_manager.cc:1120] Deleting tablet 239f14be29e54cd29bb5e96622aacd55
0503 14:12:22.175413 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.175551 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2919891}
W20240503 14:12:22.181133 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 239f14be29e54cd29bb5e96622aacd55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 239f14be29e54cd29bb5e96622aacd55 already in progress: creating tablet
I20240503 14:12:22.178126 28600 tablet_bootstrap.cc:654] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.183480 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 743) took 2930 ms. Trace:
I20240503 14:12:22.183697 28653 rpcz_store.cc:276] 0503 14:12:19.252871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.253817 (+ 946us) service_pool.cc:225] Handling call
0503 14:12:22.183206 (+2929389us) ts_tablet_manager.cc:1120] Deleting tablet cace51f8e5ff49f596c94f3698fdf9ef
0503 14:12:22.183228 (+ 22us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.183450 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2926533}
W20240503 14:12:22.184499 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet cace51f8e5ff49f596c94f3698fdf9ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cace51f8e5ff49f596c94f3698fdf9ef already in progress: creating tablet
I20240503 14:12:22.184847 28661 ts_tablet_manager.cc:1888] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:22.185171 29892 tablet.cc:1717] T e6c90869b5034128864faaf05697c230 P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:22.185186 28661 log.cc:1196] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/b25c852aa7aa4fbfafdacec43f5bc426
I20240503 14:12:22.185181 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 741) took 2932 ms. Trace:
I20240503 14:12:22.188778 28653 rpcz_store.cc:276] 0503 14:12:19.252615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.253731 (+ 1116us) service_pool.cc:225] Handling call
0503 14:12:22.185050 (+2931319us) ts_tablet_manager.cc:1120] Deleting tablet cc004243d5a947e3a8852605b0414c95
0503 14:12:22.185061 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.185165 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1296,"tablet-delete.queue_time_us":2927947}
W20240503 14:12:22.189564 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet cc004243d5a947e3a8852605b0414c95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc004243d5a947e3a8852605b0414c95 already in progress: creating tablet
I20240503 14:12:22.189858 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 742) took 2937 ms. Trace:
I20240503 14:12:22.190001 28653 rpcz_store.cc:276] 0503 14:12:19.252762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.253793 (+ 1031us) service_pool.cc:225] Handling call
0503 14:12:22.189604 (+2935811us) ts_tablet_manager.cc:1120] Deleting tablet 6809b59e87644316b9a450d6b670da2b
0503 14:12:22.189616 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.189834 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1196,"tablet-delete.queue_time_us":2932352}
I20240503 14:12:22.190681 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 744) took 2937 ms. Trace:
W20240503 14:12:22.190672 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6809b59e87644316b9a450d6b670da2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6809b59e87644316b9a450d6b670da2b already in progress: creating tablet
I20240503 14:12:22.190843 28653 rpcz_store.cc:276] 0503 14:12:19.252969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.253874 (+ 905us) service_pool.cc:225] Handling call
0503 14:12:22.190562 (+2936688us) ts_tablet_manager.cc:1120] Deleting tablet bf8d2474104a41a989cd7bd4480367f5
0503 14:12:22.190573 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.190665 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2931206}
W20240503 14:12:22.191408 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet bf8d2474104a41a989cd7bd4480367f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8d2474104a41a989cd7bd4480367f5 already in progress: creating tablet
I20240503 14:12:22.191440 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 753) took 2458 ms. Trace:
I20240503 14:12:22.191574 28653 rpcz_store.cc:276] 0503 14:12:19.733034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.733095 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:22.191334 (+2458239us) ts_tablet_manager.cc:1120] Deleting tablet 08cc84cf703048148abfe03b7dee12ce
0503 14:12:22.191344 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.191424 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457989}
W20240503 14:12:22.192155 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 08cc84cf703048148abfe03b7dee12ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cc84cf703048148abfe03b7dee12ce already in progress: creating tablet
I20240503 14:12:22.192134 28653 tablet_replica.cc:330] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.192692 28653 raft_consensus.cc:2229] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.193537 28653 raft_consensus.cc:2258] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.195289 28653 ts_tablet_manager.cc:1875] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.200963 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6331f7c5c3804f44833a92e6ded96825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:22.201786 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6331f7c5c3804f44833a92e6ded96825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:22.206405 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3ce7004b7c164629a92e6bcbb421348d" } modify_external_catalogs: true
I20240503 14:12:22.206718 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3ce7004b7c164629a92e6bcbb421348d" } modify_external_catalogs: true
I20240503 14:12:22.220551 29867 raft_consensus.cc:482] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:22.225010 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet ff07bd9bd59f4b13955904cc172c044c (table default.Table_a42d70182460481cbcf7f90a0dc57481 [id=b040f9096f95497a8c418a5e6795c21a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ff07bd9bd59f4b13955904cc172c044c consensus state has no leader
I20240503 14:12:22.243254 28653 ts_tablet_manager.cc:1888] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.243394 28653 log.cc:1196] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2f89fca0480d4bf28160384ae0421bce
I20240503 14:12:22.243697 28653 ts_tablet_manager.cc:1909] T 2f89fca0480d4bf28160384ae0421bce P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.244912 29845 raft_consensus.cc:482] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.245091 29845 raft_consensus.cc:504] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.172677 29901 consensus_queue.cc:237] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:22.250097 28427 leader_election.cc:341] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: bf21f6234dd64cf8a32f1599fe50746e
I20240503 14:12:22.251209 29845 leader_election.cc:288] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.255348 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 754) took 2521 ms. Trace:
I20240503 14:12:22.258126 28653 rpcz_store.cc:276] 0503 14:12:19.733509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.733549 (+ 40us) service_pool.cc:225] Handling call
0503 14:12:22.192055 (+2458506us) ts_tablet_manager.cc:1120] Deleting tablet 2f89fca0480d4bf28160384ae0421bce
0503 14:12:22.192064 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.228554 (+ 36490us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.243128 (+ 14574us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.255309 (+ 12181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458355}
I20240503 14:12:22.253779 28427 leader_election.cc:302] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:22.260264 28427 leader_election.cc:302] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:22.264542 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2f89fca0480d4bf28160384ae0421bce (table default.Table_5654996201694c1b90fc6e728e068ca3 [id=9a6689ece91646069be1867e8f141392]) successfully deleted
I20240503 14:12:22.265977 28653 tablet_replica.cc:330] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.266242 28653 raft_consensus.cc:2229] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.266443 28653 raft_consensus.cc:2258] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.267199 28653 ts_tablet_manager.cc:1875] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.259967 29845 raft_consensus.cc:2781] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:22.276343 29845 raft_consensus.cc:482] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3a6b9b7913444d29c96581af6febf33)
I20240503 14:12:22.276471 29845 raft_consensus.cc:3037] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:22.279906 29845 raft_consensus.cc:504] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:22.281154 29845 leader_election.cc:288] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.262145 28661 ts_tablet_manager.cc:1909] T b25c852aa7aa4fbfafdacec43f5bc426 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:22.263958 29750 raft_consensus.cc:504] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.282150 29750 leader_election.cc:288] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.272529 29916 tablet.cc:1722] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:22.273685 29915 tablet.cc:1717] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:22.275153 28604 tablet_bootstrap.cc:492] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:22.288250 28604 ts_tablet_manager.cc:1367] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.165s user 0.001s sys 0.045s
I20240503 14:12:22.289060 29915 tablet.cc:1717] T e6c90869b5034128864faaf05697c230 P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:22.289352 28604 raft_consensus.cc:348] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.289594 28604 raft_consensus.cc:374] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.289654 28604 raft_consensus.cc:729] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.289880 28604 consensus_queue.cc:260] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.290452 29750 raft_consensus.cc:2781] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.290585 29750 raft_consensus.cc:686] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:22.290843 29750 consensus_queue.cc:237] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.292469 28604 ts_tablet_manager.cc:1398] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:22.292877 28604 tablet_bootstrap.cc:492] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:22.295223 28604 tablet_bootstrap.cc:654] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.302786 28427 leader_election.cc:302] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:22.303349 28427 consensus_peers.cc:485] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:22.303730 28427 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:22.304172 29845 raft_consensus.cc:2781] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.304653 29845 raft_consensus.cc:686] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:22.305002 29845 consensus_queue.cc:237] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.312548 28600 tablet_bootstrap.cc:492] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:22.312834 28600 ts_tablet_manager.cc:1367] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.152s user 0.027s sys 0.032s
I20240503 14:12:22.314112 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 737) took 3731 ms. Trace:
I20240503 14:12:22.314348 28661 rpcz_store.cc:276] 0503 14:12:18.582159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.582473 (+ 314us) service_pool.cc:225] Handling call
0503 14:12:21.902562 (+3320089us) ts_tablet_manager.cc:1120] Deleting tablet b25c852aa7aa4fbfafdacec43f5bc426
0503 14:12:21.902810 (+ 248us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.033056 (+130246us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.184626 (+151570us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.314054 (+129428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3319416}
I20240503 14:12:22.314886 28661 tablet_replica.cc:330] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:22.316165 28600 raft_consensus.cc:348] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.316525 28600 raft_consensus.cc:374] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.316658 28600 raft_consensus.cc:729] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.316978 28600 consensus_queue.cc:260] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.317919 28600 ts_tablet_manager.cc:1398] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:12:22.318717 28614 tablet_bootstrap.cc:654] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.319413 28600 tablet_bootstrap.cc:492] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.325047 29828 ts_tablet_manager.cc:1909] T 5a081ba59ad44890bf2f94874c034277 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:22.327705 28661 raft_consensus.cc:2229] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.331578 28653 ts_tablet_manager.cc:1888] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.331773 28653 log.cc:1196] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/495190fca06a40a386c8215eb2195920
I20240503 14:12:22.333433 28653 ts_tablet_manager.cc:1909] T 495190fca06a40a386c8215eb2195920 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.338256 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 755) took 2604 ms. Trace:
I20240503 14:12:22.346854 28653 rpcz_store.cc:276] 0503 14:12:19.733809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.733846 (+ 37us) service_pool.cc:225] Handling call
0503 14:12:22.265870 (+2532024us) ts_tablet_manager.cc:1120] Deleting tablet 495190fca06a40a386c8215eb2195920
0503 14:12:22.265889 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.300520 (+ 34631us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.303978 (+ 3458us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.338221 (+ 34243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2531888}
I20240503 14:12:22.347679 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 495190fca06a40a386c8215eb2195920 (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]) successfully deleted
I20240503 14:12:22.349234 29811 raft_consensus.cc:482] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.350172 29811 raft_consensus.cc:504] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.350929 29811 leader_election.cc:288] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.351784 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:22.347801 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 756) took 2613 ms. Trace:
W20240503 14:12:22.352587 28150 leader_election.cc:341] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.352609 28653 rpcz_store.cc:276] 0503 14:12:19.734139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.734181 (+ 42us) service_pool.cc:225] Handling call
0503 14:12:22.347619 (+2613438us) ts_tablet_manager.cc:1120] Deleting tablet 2c1db04ab92c4af0847730a47635a697
0503 14:12:22.347635 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.347777 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2613289}
W20240503 14:12:22.353421 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2c1db04ab92c4af0847730a47635a697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c1db04ab92c4af0847730a47635a697 already in progress: creating tablet
I20240503 14:12:22.353653 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 757) took 2619 ms. Trace:
I20240503 14:12:22.355573 28614 tablet_bootstrap.cc:492] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.355773 28614 ts_tablet_manager.cc:1367] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.192s user 0.009s sys 0.001s
I20240503 14:12:22.355764 28653 rpcz_store.cc:276] 0503 14:12:19.734428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.734465 (+ 37us) service_pool.cc:225] Handling call
0503 14:12:22.353415 (+2618950us) ts_tablet_manager.cc:1120] Deleting tablet 37364d5facc241f4bbccc979a50a0ca8
0503 14:12:22.353428 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.353631 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2618810}
W20240503 14:12:22.356554 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 37364d5facc241f4bbccc979a50a0ca8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37364d5facc241f4bbccc979a50a0ca8 already in progress: creating tablet
I20240503 14:12:22.356600 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 758) took 2621 ms. Trace:
I20240503 14:12:22.372350 28653 rpcz_store.cc:276] 0503 14:12:19.734720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.734759 (+ 39us) service_pool.cc:225] Handling call
0503 14:12:22.356445 (+2621686us) ts_tablet_manager.cc:1120] Deleting tablet 03700c1efc08471cb67dfe3a8016a740
0503 14:12:22.356456 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.356580 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621555}
W20240503 14:12:22.373078 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 03700c1efc08471cb67dfe3a8016a740 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03700c1efc08471cb67dfe3a8016a740 already in progress: creating tablet
I20240503 14:12:22.373319 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 759) took 2638 ms. Trace:
I20240503 14:12:22.373381 29932 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:12:22.373453 28653 rpcz_store.cc:276] 0503 14:12:19.735004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.735039 (+ 35us) service_pool.cc:225] Handling call
0503 14:12:22.373039 (+2638000us) ts_tablet_manager.cc:1120] Deleting tablet 6de9e159d2ba47039dd6bd207a9cddeb
0503 14:12:22.373054 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.373293 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2637866}
W20240503 14:12:22.374042 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6de9e159d2ba47039dd6bd207a9cddeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6de9e159d2ba47039dd6bd207a9cddeb already in progress: creating tablet
I20240503 14:12:22.374383 28653 tablet_replica.cc:330] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.374636 28653 raft_consensus.cc:2229] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.375121 28653 raft_consensus.cc:2258] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.375527 28614 raft_consensus.cc:348] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.375746 28614 raft_consensus.cc:374] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.375816 28614 raft_consensus.cc:729] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.376102 28614 consensus_queue.cc:260] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.376684 28614 ts_tablet_manager.cc:1398] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20240503 14:12:22.377002 28614 tablet_bootstrap.cc:492] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.378491 28614 tablet_bootstrap.cc:654] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.379758 29932 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:12:22.384289 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 802 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:12:22.384316 28898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 802 } } ignore_live_leader: false dest_uuid: "bb6a7aee99214df4b02858ddae025b14" is_pre_election: true
I20240503 14:12:22.385156 29436 leader_election.cc:302] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc; no voters: bb6a7aee99214df4b02858ddae025b14, c0690a516c89446d85c88cdf64fa4238
I20240503 14:12:22.386812 29932 leader_election.cc:288] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:12:22.387566 29828 tablet_replica.cc:330] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:22.387542 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 5a081ba59ad44890bf2f94874c034277 (table default.Table_1145b045780c4d1e8fedd333eea661d4 [id=114dfd39b4ea4958a08161a34e4a1f9f]) successfully deleted
I20240503 14:12:22.388397 29932 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:22.391197 28653 ts_tablet_manager.cc:1875] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.328114 28661 raft_consensus.cc:2258] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.403939 28661 ts_tablet_manager.cc:1875] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.411680 28600 tablet_bootstrap.cc:654] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.255187 29510 rpcz_store.cc:276] 0503 14:12:16.125197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:16.125286 (+ 89us) service_pool.cc:225] Handling call
0503 14:12:19.943833 (+3818547us) raft_consensus.cc:1457] Updating replica for 400 ops
0503 14:12:20.049706 (+105873us) raft_consensus.cc:1520] Early marking committed up to index 0
0503 14:12:20.049717 (+ 11us) raft_consensus.cc:1525] Triggering prepare for 400 ops
0503 14:12:20.053074 (+ 3357us) write_op.cc:270] Start()
0503 14:12:20.053175 (+ 101us) write_op.cc:276] Timestamp: P: 1714745528049819 usec, L: 0
0503 14:12:20.153017 (+ 99842us) write_op.cc:270] Start()
0503 14:12:20.153058 (+ 41us) write_op.cc:276] Timestamp: P: 1714745528091847 usec, L: 0
0503 14:12:20.153450 (+ 392us) write_op.cc:270] Start()
0503 14:12:20.153505 (+ 55us) write_op.cc:276] Timestamp: P: 1714745528106945 usec, L: 0
0503 14:12:20.153693 (+ 188us) write_op.cc:270] Start()
0503 14:12:20.153713 (+ 20us) write_op.cc:276] Timestamp: P: 1714745528114097 usec, L: 0
0503 14:12:20.153858 (+ 145us) write_op.cc:270] Start()
0503 14:12:20.153878 (+ 20us) write_op.cc:276] Timestamp: P: 1714745528162728 usec, L: 0
0503 14:12:20.154005 (+ 127us) write_op.cc:270] Start()
0503 14:12:20.154024 (+ 19us) write_op.cc:276] Timestamp: P: 1714745528171318 usec, L: 0
0503 14:12:20.154222 (+ 198us) write_op.cc:270] Start()
0503 14:12:20.154297 (+ 75us) write_op.cc:276] Timestamp: P: 1714745528179225 usec, L: 0
0503 14:12:20.154510 (+ 213us) write_op.cc:270] Start()
0503 14:12:20.154531 (+ 21us) write_op.cc:276] Timestamp: P: 1714745528182841 usec, L: 0
0503 14:12:20.154663 (+ 132us) write_op.cc:270] Start()
0503 14:12:20.154681 (+ 18us) write_op.cc:276] Timestamp: P: 1714745528199060 usec, L: 0
0503 14:12:20.154805 (+ 124us) write_op.cc:270] Start()
0503 14:12:20.154824 (+ 19us) write_op.cc:276] Timestamp: P: 1714745528222393 usec, L: 0
0503 14:12:20.155018 (+ 194us) write_op.cc:270] Start()
0503 14:12:20.155039 (+ 21us) write_op.cc:276] Timestamp: P: 1714745528225587 usec, L: 0
0503 14:12:20.155178 (+ 139us) write_op.cc:270] Start()
0503 14:12:20.155196 (+ 18us) write_op.cc:276] Timestamp: P: 1714745528228279 usec, L: 0
0503 14:12:20.155352 (+ 156us) write_op.cc:270] Start()
0503 14:12:20.155371 (+ 19us) write_op.cc:276] Timestamp: P: 1714745528229136 usec, L: 0
0503 14:12:20.155480 (+ 109us) write_op.cc:270] Start()
0503 14:12:20.180723 (+ 25243us) write_op.cc:276] Timestamp: P: 1714745528234878 usec, L: 0
0503 14:12:20.181096 (+ 373us) write_op.cc:270] Start()
0503 14:12:20.181125 (+ 29us) write_op.cc:276] Timestamp: P: 1714745528236000 usec, L: 0
0503 14:12:20.181274 (+ 149us) write_op.cc:270] Start()
0503 14:12:20.181293 (+ 19us) write_op.cc:276] Timestamp: P: 1714745528236823 usec, L: 0
0503 14:12:20.181483 (+ 190us) write_op.cc:270] Start()
0503 14:12:20.181515 (+ 32us) write_op.cc:276] Timestamp: P: 1714745528437277 usec, L: 0
0503 14:12:20.181713 (+ 198us) write_op.cc:270] Start()
0503 14:12:20.181733 (+ 20us) write_op.cc:276] Timestamp: P: 1714745528458599 usec, L: 0
0503 14:12:20.181893 (+ 160us) write_op.cc:270] Start()
0503 14:12:20.181912 (+ 19us) write_op.cc:276] Timestamp: P: 1714745528460724 usec, L: 0
0503 14:12:20.216358 (+ 34446us) write_op.cc:270] Start()
0503 14:12:20.216412 (+ 54us) write_op.cc:276] Timestamp: P: 1714745528463162 usec, L: 0
0503 14:12:20.216700 (+ 288us) write_op.cc:270] Start()
0503 14:12:20.216727 (+ 27us) write_op.cc:276] Timestamp: P: 1714745528465062 usec, L: 0
0503 14:12:20.216873 (+ 146us) write_op.cc:270] Start()
0503 14:12:20.216894 (+ 21us) write_op.cc:276] Timestamp: P: 1714745528471110 usec, L: 0
0503 14:12:20.217073 (+ 179us) write_op.cc:270] Start()
0503 14:12:20.217104 (+ 31us) write_op.cc:276] Timestamp: P: 1714745528475691 usec, L: 0
0503 14:12:20.217334 (+ 230us) write_op.cc:270] Start()
0503 14:12:20.217358 (+ 24us) write_op.cc:276] Timestamp: P: 1714745528478980 usec, L: 0
0503 14:12:20.217535 (+ 177us) write_op.cc:270] Start()
0503 14:12:20.217557 (+ 22us) write_op.cc:276] Timestamp: P: 1714745528674685 usec, L: 0
0503 14:12:20.217719 (+ 162us) write_op.cc:270] Start()
0503 14:12:20.217742 (+ 23us) write_op.cc:276] Timestamp: P: 1714745528679048 usec, L: 0
0503 14:12:20.217904 (+ 162us) write_op.cc:270] Start()
0503 14:12:20.217924 (+ 20us) write_op.cc:276] Timestamp: P: 1714745528681071 usec, L: 0
0503 14:12:20.218039 (+ 115us) write_op.cc:270] Start()
0503 14:12:20.218056 (+ 17us) write_op.cc:276] Timestamp: P: 1714745528711393 usec, L: 0
0503 14:12:20.218212 (+ 156us) write_op.cc:270] Start()
0503 14:12:20.218232 (+ 20us) write_op.cc:276] Timestamp: P: 1714745528712320 usec, L: 0
0503 14:12:20.218383 (+ 151us) write_op.cc:270] Start()
0503 14:12:20.218403 (+ 20us) write_op.cc:276] Timestamp: P: 1714745528713071 usec, L: 0
0503 14:12:20.218537 (+ 134us) write_op.cc:270] Start()
0503 14:12:20.218556 (+ 19us) write_op.cc:276] Timestamp: P: 1714745528713787 usec, L: 0
0503 14:12:20.218704 (+ 148us) write_op.cc:270] Start()
0503 14:12:20.218725 (+ 21us) write_op.cc:276] Timestamp: P: 1714745528728304 usec, L: 0
0503 14:12:20.218843 (+ 118us) write_op.cc:270] Start()
0503 14:12:20.218877 (+ 34us) write_op.cc:276] Timestamp: P: 1714745529294782 usec, L: 0
0503 14:12:20.218993 (+ 116us) write_op.cc:270] Start()
0503 14:12:20.219010 (+ 17us) write_op.cc:276] Timestamp: P: 1714745529337094 usec, L: 0
0503 14:12:20.219162 (+ 152us) write_op.cc:270] Start()
0503 14:12:20.219181 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529344399 usec, L: 0
0503 14:12:20.219330 (+ 149us) write_op.cc:270] Start()
0503 14:12:20.219347 (+ 17us) write_op.cc:276] Timestamp: P: 1714745529345960 usec, L: 0
0503 14:12:20.219469 (+ 122us) write_op.cc:270] Start()
0503 14:12:20.219499 (+ 30us) write_op.cc:276] Timestamp: P: 1714745529363703 usec, L: 0
0503 14:12:20.219649 (+ 150us) write_op.cc:270] Start()
0503 14:12:20.219670 (+ 21us) write_op.cc:276] Timestamp: P: 1714745529364559 usec, L: 0
0503 14:12:20.219853 (+ 183us) write_op.cc:270] Start()
0503 14:12:20.219872 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529365226 usec, L: 0
0503 14:12:20.219998 (+ 126us) write_op.cc:270] Start()
0503 14:12:20.220016 (+ 18us) write_op.cc:276] Timestamp: P: 1714745529365827 usec, L: 0
0503 14:12:20.220196 (+ 180us) write_op.cc:270] Start()
0503 14:12:20.220218 (+ 22us) write_op.cc:276] Timestamp: P: 1714745529440438 usec, L: 0
0503 14:12:20.220380 (+ 162us) write_op.cc:270] Start()
0503 14:12:20.220399 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529528654 usec, L: 0
0503 14:12:20.220532 (+ 133us) write_op.cc:270] Start()
0503 14:12:20.220551 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529548905 usec, L: 0
0503 14:12:20.220696 (+ 145us) write_op.cc:270] Start()
0503 14:12:20.220715 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529549513 usec, L: 0
0503 14:12:20.220867 (+ 152us) write_op.cc:270] Start()
0503 14:12:20.220886 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529550020 usec, L: 0
0503 14:12:20.221000 (+ 114us) write_op.cc:270] Start()
0503 14:12:20.221020 (+ 20us) write_op.cc:276] Timestamp: P: 1714745529556933 usec, L: 0
0503 14:12:20.221175 (+ 155us) write_op.cc:270] Start()
0503 14:12:20.221196 (+ 21us) write_op.cc:276] Timestamp: P: 1714745529574657 usec, L: 0
0503 14:12:20.221347 (+ 151us) write_op.cc:270] Start()
0503 14:12:20.221369 (+ 22us) write_op.cc:276] Timestamp: P: 1714745529575373 usec, L: 0
0503 14:12:20.221519 (+ 150us) write_op.cc:270] Start()
0503 14:12:20.221539 (+ 20us) write_op.cc:276] Timestamp: P: 1714745529594205 usec, L: 0
0503 14:12:20.221732 (+ 193us) write_op.cc:270] Start()
0503 14:12:20.221754 (+ 22us) write_op.cc:276] Timestamp: P: 1714745529598293 usec, L: 0
0503 14:12:20.222026 (+ 272us) write_op.cc:270] Start()
0503 14:12:20.222049 (+ 23us) write_op.cc:276] Timestamp: P: 1714745529600794 usec, L: 0
0503 14:12:20.222197 (+ 148us) write_op.cc:270] Start()
0503 14:12:20.222219 (+ 22us) write_op.cc:276] Timestamp: P: 1714745529627161 usec, L: 0
0503 14:12:20.222401 (+ 182us) write_op.cc:270] Start()
0503 14:12:20.222422 (+ 21us) write_op.cc:276] Timestamp: P: 1714745529629650 usec, L: 0
0503 14:12:20.222549 (+ 127us) write_op.cc:270] Start()
0503 14:12:20.222568 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529631913 usec, L: 0
0503 14:12:20.222733 (+ 165us) write_op.cc:270] Start()
0503 14:12:20.222753 (+ 20us) write_op.cc:276] Timestamp: P: 1714745529632725 usec, L: 0
0503 14:12:20.222886 (+ 133us) write_op.cc:270] Start()
0503 14:12:20.222903 (+ 17us) write_op.cc:276] Timestamp: P: 1714745529633991 usec, L: 0
0503 14:12:20.223020 (+ 117us) write_op.cc:270] Start()
0503 14:12:20.223037 (+ 17us) write_op.cc:276] Timestamp: P: 1714745529634756 usec, L: 0
0503 14:12:20.223201 (+ 164us) write_op.cc:270] Start()
0503 14:12:20.223219 (+ 18us) write_op.cc:276] Timestamp: P: 1714745529653948 usec, L: 0
0503 14:12:20.223346 (+ 127us) write_op.cc:270] Start()
0503 14:12:20.223362 (+ 16us) write_op.cc:276] Timestamp: P: 1714745529658543 usec, L: 0
0503 14:12:20.223478 (+ 116us) write_op.cc:270] Start()
0503 14:12:20.223526 (+ 48us) write_op.cc:276] Timestamp: P: 1714745529659697 usec, L: 0
0503 14:12:20.223655 (+ 129us) write_op.cc:270] Start()
0503 14:12:20.223674 (+ 19us) write_op.cc:276] Timestamp: P: 1714745529940357 usec, L: 0
0503 14:12:20.223796 (+ 122us) write_op.cc:270] Start()
0503 14:12:20.223812 (+ 16us) write_op.cc:276] Timestamp: P: 1714745529941135 usec, L: 0
0503 14:12:20.223925 (+ 113us) write_op.cc:270] Start()
0503 14:12:20.223945 (+ 20us) write_op.cc:276] Timestamp: P: 1714745529941765 usec, L: 0
0503 14:12:20.236600 (+ 12655us) write_op.cc:270] Start()
0503 14:12:20.236662 (+ 62us) write_op.cc:276] Timestamp: P: 1714745529949463 usec, L: 0
0503 14:12:20.237056 (+ 394us) write_op.cc:270] Start()
0503 14:12:20.237129 (+ 73us) write_op.cc:276] Timestamp: P: 1714745529951541 usec, L: 0
0503 14:12:20.237363 (+ 234us) write_op.cc:270] Start()
0503 14:12:20.237388 (+ 25us) write_op.cc:276] Timestamp: P: 1714745529980261 usec, L: 0
0503 14:12:20.237569 (+ 181us) write_op.cc:270] Start()
0503 14:12:20.237591 (+ 22us) write_op.cc:276] Timestamp: P: 1714745529981069 usec, L: 0
0503 14:12:20.237721 (+ 130us) write_op.cc:270] Start()
0503 14:12:20.237739 (+ 18us) write_op.cc:276] Timestamp: P: 1714745529981659 usec, L: 0
0503 14:12:20.237880 (+ 141us) write_op.cc:270] Start()
0503 14:12:20.237921 (+ 41us) write_op.cc:276] Timestamp: P: 1714745529982243 usec, L: 0
0503 14:12:20.238082 (+ 161us) write_op.cc:270] Start()
0503 14:12:20.238114 (+ 32us) write_op.cc:276] Timestamp: P: 1714745529983822 usec, L: 0
0503 14:12:20.238323 (+ 209us) write_op.cc:270] Start()
0503 14:12:20.238343 (+ 20us) write_op.cc:276] Timestamp: P: 1714745529988176 usec, L: 0
0503 14:12:20.238504 (+ 161us) write_op.cc:270] Start()
0503 14:12:20.238524 (+ 20us) write_op.cc:276] Timestamp: P: 1714745529991389 usec, L: 0
0503 14:12:20.238672 (+ 148us) write_op.cc:270] Start()
0503 14:12:20.238694 (+ 22us) write_op.cc:276] Timestamp: P: 1714745529992961 usec, L: 0
0503 14:12:20.247386 (+ 8692us) write_op.cc:270] Start()
0503 14:12:20.247439 (+ 53us) write_op.cc:276] Timestamp: P: 1714745530021449 usec, L: 0
0503 14:12:20.247923 (+ 484us) write_op.cc:270] Start()
0503 14:12:20.247958 (+ 35us) write_op.cc:276] Timestamp: P: 1714745530022241 usec, L: 0
0503 14:12:20.248663 (+ 705us) write_op.cc:270] Start()
0503 14:12:20.248698 (+ 35us) write_op.cc:276] Timestamp: P: 1714745530023432 usec, L: 0
0503 14:12:20.248924 (+ 226us) write_op.cc:270] Start()
0503 14:12:20.248949 (+ 25us) write_op.cc:276] Timestamp: P: 1714745530115574 usec, L: 0
0503 14:12:20.249123 (+ 174us) write_op.cc:270] Start()
0503 14:12:20.249145 (+ 22us) write_op.cc:276] Timestamp: P: 1714745530130204 usec, L: 0
0503 14:12:20.249294 (+ 149us) write_op.cc:270] Start()
0503 14:12:20.249337 (+ 43us) write_op.cc:276] Timestamp: P: 1714745530177396 usec, L: 0
0503 14:12:20.249501 (+ 164us) write_op.cc:270] Start()
0503 14:12:20.249523 (+ 22us) write_op.cc:276] Timestamp: P: 1714745530178092 usec, L: 0
0503 14:12:20.250799 (+ 1276us) write_op.cc:270] Start()
0503 14:12:20.250831 (+ 32us) write_op.cc:276] Timestamp: P: 1714745530246168 usec, L: 0
0503 14:12:20.251040 (+ 209us) write_op.cc:270] Start()
0503 14:12:20.251065 (+ 25us) write_op.cc:276] Timestamp: P: 1714745530247370 usec, L: 0
0503 14:12:20.251311 (+ 246us) write_op.cc:270] Start()
0503 14:12:20.251337 (+ 26us) write_op.cc:276] Timestamp: P: 1714745530248871 usec, L: 0
0503 14:12:20.251479 (+ 142us) write_op.cc:270] Start()
0503 14:12:20.251500 (+ 21us) write_op.cc:276] Timestamp: P: 1714745530250002 usec, L: 0
0503 14:12:20.251676 (+ 176us) write_op.cc:270] Start()
0503 14:12:20.251699 (+ 23us) write_op.cc:276] Timestamp: P: 1714745530251023 usec, L: 0
0503 14:12:20.251822 (+ 123us) write_op.cc:270] Start()
0503 14:12:20.251841 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530251900 usec, L: 0
0503 14:12:20.252018 (+ 177us) write_op.cc:270] Start()
0503 14:12:20.252041 (+ 23us) write_op.cc:276] Timestamp: P: 1714745530252849 usec, L: 0
0503 14:12:20.252249 (+ 208us) write_op.cc:270] Start()
0503 14:12:20.252272 (+ 23us) write_op.cc:276] Timestamp: P: 1714745530253711 usec, L: 0
0503 14:12:20.252397 (+ 125us) write_op.cc:270] Start()
0503 14:12:20.252415 (+ 18us) write_op.cc:276] Timestamp: P: 1714745530263663 usec, L: 0
0503 14:12:20.252547 (+ 132us) write_op.cc:270] Start()
0503 14:12:20.252567 (+ 20us) write_op.cc:276] Timestamp: P: 1714745530280839 usec, L: 0
0503 14:12:20.252716 (+ 149us) write_op.cc:270] Start()
0503 14:12:20.252734 (+ 18us) write_op.cc:276] Timestamp: P: 1714745530281511 usec, L: 0
0503 14:12:20.252910 (+ 176us) write_op.cc:270] Start()
0503 14:12:20.252929 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530283278 usec, L: 0
0503 14:12:20.253166 (+ 237us) write_op.cc:270] Start()
0503 14:12:20.253189 (+ 23us) write_op.cc:276] Timestamp: P: 1714745530283917 usec, L: 0
0503 14:12:20.256911 (+ 3722us) write_op.cc:270] Start()
0503 14:12:20.256952 (+ 41us) write_op.cc:276] Timestamp: P: 1714745530284635 usec, L: 0
0503 14:12:20.257170 (+ 218us) write_op.cc:270] Start()
0503 14:12:20.257194 (+ 24us) write_op.cc:276] Timestamp: P: 1714745530286119 usec, L: 0
0503 14:12:20.257449 (+ 255us) write_op.cc:270] Start()
0503 14:12:20.257471 (+ 22us) write_op.cc:276] Timestamp: P: 1714745530286791 usec, L: 0
0503 14:12:20.257612 (+ 141us) write_op.cc:270] Start()
0503 14:12:20.257632 (+ 20us) write_op.cc:276] Timestamp: P: 1714745530290701 usec, L: 0
0503 14:12:20.257786 (+ 154us) write_op.cc:270] Start()
0503 14:12:20.257807 (+ 21us) write_op.cc:276] Timestamp: P: 1714745530298626 usec, L: 0
0503 14:12:20.257985 (+ 178us) write_op.cc:270] Start()
0503 14:12:20.258009 (+ 24us) write_op.cc:276] Timestamp: P: 1714745530303825 usec, L: 0
0503 14:12:20.258147 (+ 138us) write_op.cc:270] Start()
0503 14:12:20.258165 (+ 18us) write_op.cc:276] Timestamp: P: 1714745530350630 usec, L: 0
0503 14:12:20.258379 (+ 214us) write_op.cc:270] Start()
0503 14:12:20.258402 (+ 23us) write_op.cc:276] Timestamp: P: 1714745530385689 usec, L: 0
0503 14:12:20.260939 (+ 2537us) write_op.cc:270] Start()
0503 14:12:20.260971 (+ 32us) write_op.cc:276] Timestamp: P: 1714745530409235 usec, L: 0
0503 14:12:20.261162 (+ 191us) write_op.cc:270] Start()
0503 14:12:20.261182 (+ 20us) write_op.cc:276] Timestamp: P: 1714745530444601 usec, L: 0
0503 14:12:20.261302 (+ 120us) write_op.cc:270] Start()
0503 14:12:20.261318 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530466046 usec, L: 0
0503 14:12:20.261426 (+ 108us) write_op.cc:270] Start()
0503 14:12:20.261441 (+ 15us) write_op.cc:276] Timestamp: P: 1714745530479996 usec, L: 0
0503 14:12:20.261530 (+ 89us) write_op.cc:270] Start()
0503 14:12:20.261546 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530480782 usec, L: 0
0503 14:12:20.261654 (+ 108us) write_op.cc:270] Start()
0503 14:12:20.261670 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530481530 usec, L: 0
0503 14:12:20.261764 (+ 94us) write_op.cc:270] Start()
0503 14:12:20.261778 (+ 14us) write_op.cc:276] Timestamp: P: 1714745530520315 usec, L: 0
0503 14:12:20.261889 (+ 111us) write_op.cc:270] Start()
0503 14:12:20.261907 (+ 18us) write_op.cc:276] Timestamp: P: 1714745530529897 usec, L: 0
0503 14:12:20.262019 (+ 112us) write_op.cc:270] Start()
0503 14:12:20.262035 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530538498 usec, L: 0
0503 14:12:20.262169 (+ 134us) write_op.cc:270] Start()
0503 14:12:20.262189 (+ 20us) write_op.cc:276] Timestamp: P: 1714745530548138 usec, L: 0
0503 14:12:20.262345 (+ 156us) write_op.cc:270] Start()
0503 14:12:20.262366 (+ 21us) write_op.cc:276] Timestamp: P: 1714745530549876 usec, L: 0
0503 14:12:20.262487 (+ 121us) write_op.cc:270] Start()
0503 14:12:20.262512 (+ 25us) write_op.cc:276] Timestamp: P: 1714745530551377 usec, L: 0
0503 14:12:20.262631 (+ 119us) write_op.cc:270] Start()
0503 14:12:20.262650 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530553271 usec, L: 0
0503 14:12:20.262796 (+ 146us) write_op.cc:270] Start()
0503 14:12:20.262817 (+ 21us) write_op.cc:276] Timestamp: P: 1714745530556501 usec, L: 0
0503 14:12:20.262956 (+ 139us) write_op.cc:270] Start()
0503 14:12:20.262976 (+ 20us) write_op.cc:276] Timestamp: P: 1714745530559969 usec, L: 0
0503 14:12:20.263146 (+ 170us) write_op.cc:270] Start()
0503 14:12:20.263167 (+ 21us) write_op.cc:276] Timestamp: P: 1714745530563121 usec, L: 0
0503 14:12:20.263298 (+ 131us) write_op.cc:270] Start()
0503 14:12:20.263314 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530566956 usec, L: 0
0503 14:12:20.263434 (+ 120us) write_op.cc:270] Start()
0503 14:12:20.263453 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530570307 usec, L: 0
0503 14:12:20.263589 (+ 136us) write_op.cc:270] Start()
0503 14:12:20.263608 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530601680 usec, L: 0
0503 14:12:20.263744 (+ 136us) write_op.cc:270] Start()
0503 14:12:20.263764 (+ 20us) write_op.cc:276] Timestamp: P: 1714745530621814 usec, L: 0
0503 14:12:20.263931 (+ 167us) write_op.cc:270] Start()
0503 14:12:20.263954 (+ 23us) write_op.cc:276] Timestamp: P: 1714745530637996 usec, L: 0
0503 14:12:20.264170 (+ 216us) write_op.cc:270] Start()
0503 14:12:20.264197 (+ 27us) write_op.cc:276] Timestamp: P: 1714745530658970 usec, L: 0
0503 14:12:20.264394 (+ 197us) write_op.cc:270] Start()
0503 14:12:20.264416 (+ 22us) write_op.cc:276] Timestamp: P: 1714745530677259 usec, L: 0
0503 14:12:20.264602 (+ 186us) write_op.cc:270] Start()
0503 14:12:20.264627 (+ 25us) write_op.cc:276] Timestamp: P: 1714745530686463 usec, L: 0
0503 14:12:20.264831 (+ 204us) write_op.cc:270] Start()
0503 14:12:20.264855 (+ 24us) write_op.cc:276] Timestamp: P: 1714745530687096 usec, L: 0
0503 14:12:20.265011 (+ 156us) write_op.cc:270] Start()
0503 14:12:20.265033 (+ 22us) write_op.cc:276] Timestamp: P: 1714745530710362 usec, L: 0
0503 14:12:20.265252 (+ 219us) write_op.cc:270] Start()
0503 14:12:20.265276 (+ 24us) write_op.cc:276] Timestamp: P: 1714745530712131 usec, L: 0
0503 14:12:20.265462 (+ 186us) write_op.cc:270] Start()
0503 14:12:20.265481 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530716485 usec, L: 0
0503 14:12:20.265583 (+ 102us) write_op.cc:270] Start()
0503 14:12:20.265598 (+ 15us) write_op.cc:276] Timestamp: P: 1714745530729926 usec, L: 0
0503 14:12:20.265740 (+ 142us) write_op.cc:270] Start()
0503 14:12:20.265790 (+ 50us) write_op.cc:276] Timestamp: P: 1714745530767765 usec, L: 0
0503 14:12:20.265892 (+ 102us) write_op.cc:270] Start()
0503 14:12:20.265908 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530769708 usec, L: 0
0503 14:12:20.266023 (+ 115us) write_op.cc:270] Start()
0503 14:12:20.266041 (+ 18us) write_op.cc:276] Timestamp: P: 1714745530811690 usec, L: 0
0503 14:12:20.266185 (+ 144us) write_op.cc:270] Start()
0503 14:12:20.266216 (+ 31us) write_op.cc:276] Timestamp: P: 1714745530817629 usec, L: 0
0503 14:12:20.266328 (+ 112us) write_op.cc:270] Start()
0503 14:12:20.266345 (+ 17us) write_op.cc:276] Timestamp: P: 1714745530818435 usec, L: 0
0503 14:12:20.266474 (+ 129us) write_op.cc:270] Start()
0503 14:12:20.266490 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530819037 usec, L: 0
0503 14:12:20.266606 (+ 116us) write_op.cc:270] Start()
0503 14:12:20.266622 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530822760 usec, L: 0
0503 14:12:20.266719 (+ 97us) write_op.cc:270] Start()
0503 14:12:20.266733 (+ 14us) write_op.cc:276] Timestamp: P: 1714745530824132 usec, L: 0
0503 14:12:20.266850 (+ 117us) write_op.cc:270] Start()
0503 14:12:20.266868 (+ 18us) write_op.cc:276] Timestamp: P: 1714745530825290 usec, L: 0
0503 14:12:20.266997 (+ 129us) write_op.cc:270] Start()
0503 14:12:20.267015 (+ 18us) write_op.cc:276] Timestamp: P: 1714745530847336 usec, L: 0
0503 14:12:20.267179 (+ 164us) write_op.cc:270] Start()
0503 14:12:20.267198 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530918603 usec, L: 0
0503 14:12:20.267334 (+ 136us) write_op.cc:270] Start()
0503 14:12:20.267353 (+ 19us) write_op.cc:276] Timestamp: P: 1714745530921293 usec, L: 0
0503 14:12:20.267455 (+ 102us) write_op.cc:270] Start()
0503 14:12:20.267470 (+ 15us) write_op.cc:276] Timestamp: P: 1714745530924551 usec, L: 0
0503 14:12:20.267646 (+ 176us) write_op.cc:270] Start()
0503 14:12:20.267663 (+ 17us) write_op.cc:276] Timestamp: P: 1714745530925165 usec, L: 0
0503 14:12:20.267801 (+ 138us) write_op.cc:270] Start()
0503 14:12:20.267817 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530929822 usec, L: 0
0503 14:12:20.267951 (+ 134us) write_op.cc:270] Start()
0503 14:12:20.267967 (+ 16us) write_op.cc:276] Timestamp: P: 1714745530931808 usec, L: 0
0503 14:12:20.275606 (+ 7639us) write_op.cc:270] Start()
0503 14:12:20.275670 (+ 64us) write_op.cc:276] Timestamp: P: 1714745530932768 usec, L: 0
0503 14:12:20.275959 (+ 289us) write_op.cc:270] Start()
0503 14:12:20.275989 (+ 30us) write_op.cc:276] Timestamp: P: 1714745530935648 usec, L: 0
0503 14:12:20.276262 (+ 273us) write_op.cc:270] Start()
0503 14:12:20.276289 (+ 27us) write_op.cc:276] Timestamp: P: 1714745530980747 usec, L: 0
0503 14:12:20.276450 (+ 161us) write_op.cc:270] Start()
0503 14:12:20.276472 (+ 22us) write_op.cc:276] Timestamp: P: 1714745531055841 usec, L: 0
0503 14:12:20.276610 (+ 138us) write_op.cc:270] Start()
0503 14:12:20.276630 (+ 20us) write_op.cc:276] Timestamp: P: 1714745531060585 usec, L: 0
0503 14:12:20.276809 (+ 179us) write_op.cc:270] Start()
0503 14:12:20.276829 (+ 20us) write_op.cc:276] Timestamp: P: 1714745531065144 usec, L: 0
0503 14:12:20.276952 (+ 123us) write_op.cc:270] Start()
0503 14:12:20.276971 (+ 19us) write_op.cc:276] Timestamp: P: 1714745531075356 usec, L: 0
0503 14:12:20.277093 (+ 122us) write_op.cc:270] Start()
0503 14:12:20.277109 (+ 16us) write_op.cc:276] Timestamp: P: 1714745531076961 usec, L: 0
0503 14:12:20.277248 (+ 139us) write_op.cc:270] Start()
0503 14:12:20.277266 (+ 18us) write_op.cc:276] Timestamp: P: 1714745531091070 usec, L: 0
0503 14:12:20.277395 (+ 129us) write_op.cc:270] Start()
0503 14:12:20.277412 (+ 17us) write_op.cc:276] Timestamp: P: 1714745531124340 usec, L: 0
0503 14:12:20.277606 (+ 194us) write_op.cc:270] Start()
0503 14:12:20.277629 (+ 23us) write_op.cc:276] Timestamp: P: 1714745531136159 usec, L: 0
0503 14:12:20.277792 (+ 163us) write_op.cc:270] Start()
0503 14:12:20.277814 (+ 22us) write_op.cc:276] Timestamp: P: 1714745531577487 usec, L: 0
0503 14:12:20.277929 (+ 115us) write_op.cc:270] Start()
0503 14:12:20.277945 (+ 16us) write_op.cc:276] Timestamp: P: 1714745531581048 usec, L: 0
0503 14:12:20.278072 (+ 127us) write_op.cc:270] Start()
0503 14:12:20.278100 (+ 28us) write_op.cc:276] Timestamp: P: 1714745531604370 usec, L: 0
0503 14:12:20.278237 (+ 137us) write_op.cc:270] Start()
0503 14:12:20.278255 (+ 18us) write_op.cc:276] Timestamp: P: 1714745531734980 usec, L: 0
0503 14:12:20.278363 (+ 108us) write_op.cc:270] Start()
0503 14:12:20.278381 (+ 18us) write_op.cc:276] Timestamp: P: 1714745531806937 usec, L: 0
0503 14:12:20.278548 (+ 167us) write_op.cc:270] Start()
0503 14:12:20.278565 (+ 17us) write_op.cc:276] Timestamp: P: 1714745531825341 usec, L: 0
0503 14:12:20.278674 (+ 109us) write_op.cc:270] Start()
0503 14:12:20.278691 (+ 17us) write_op.cc:276] Timestamp: P: 1714745531845843 usec, L: 0
0503 14:12:20.278864 (+ 173us) write_op.cc:270] Start()
0503 14:12:20.278886 (+ 22us) write_op.cc:276] Timestamp: P: 1714745531861068 usec, L: 0
0503 14:12:20.279031 (+ 145us) write_op.cc:270] Start()
0503 14:12:20.279066 (+ 35us) write_op.cc:276] Timestamp: P: 1714745531878080 usec, L: 0
0503 14:12:20.279252 (+ 186us) write_op.cc:270] Start()
0503 14:12:20.279335 (+ 83us) write_op.cc:276] Timestamp: P: 1714745531917311 usec, L: 0
0503 14:12:20.279522 (+ 187us) write_op.cc:270] Start()
0503 14:12:20.279543 (+ 21us) write_op.cc:276] Timestamp: P: 1714745531924230 usec, L: 0
0503 14:12:20.279693 (+ 150us) write_op.cc:270] Start()
0503 14:12:20.279711 (+ 18us) write_op.cc:276] Timestamp: P: 1714745531926281 usec, L: 0
0503 14:12:20.279843 (+ 132us) write_op.cc:270] Start()
0503 14:12:20.279861 (+ 18us) write_op.cc:276] Timestamp: P: 1714745531928811 usec, L: 0
0503 14:12:20.279995 (+ 134us) write_op.cc:270] Start()
0503 14:12:20.280012 (+ 17us) write_op.cc:276] Timestamp: P: 1714745531930390 usec, L: 0
0503 14:12:20.285824 (+ 5812us) write_op.cc:270] Start()
0503 14:12:20.286763 (+ 939us) write_op.cc:276] Timestamp: P: 1714745531935031 usec, L: 0
0503 14:12:20.287081 (+ 318us) write_op.cc:270] Start()
0503 14:12:20.287119 (+ 38us) write_op.cc:276] Timestamp: P: 1714745531935843 usec, L: 0
0503 14:12:20.287318 (+ 199us) write_op.cc:270] Start()
0503 14:12:20.287344 (+ 26us) write_op.cc:276] Timestamp: P: 1714745531937320 usec, L: 0
0503 14:12:20.287503 (+ 159us) write_op.cc:270] Start()
0503 14:12:20.288549 (+ 1046us) write_op.cc:276] Timestamp: P: 1714745532004326 usec, L: 0
0503 14:12:20.288782 (+ 233us) write_op.cc:270] Start()
0503 14:12:20.288808 (+ 26us) write_op.cc:276] Timestamp: P: 1714745532046174 usec, L: 0
0503 14:12:20.288982 (+ 174us) write_op.cc:270] Start()
0503 14:12:20.289007 (+ 25us) write_op.cc:276] Timestamp: P: 1714745532052463 usec, L: 0
0503 14:12:20.289155 (+ 148us) write_op.cc:270] Start()
0503 14:12:20.289174 (+ 19us) write_op.cc:276] Timestamp: P: 1714745532053121 usec, L: 0
0503 14:12:20.289310 (+ 136us) write_op.cc:270] Start()
0503 14:12:20.289330 (+ 20us) write_op.cc:276] Timestamp: P: 1714745532056549 usec, L: 0
0503 14:12:20.290785 (+ 1455us) write_op.cc:270] Start()
0503 14:12:20.290821 (+ 36us) write_op.cc:276] Timestamp: P: 1714745532060349 usec, L: 0
0503 14:12:20.291050 (+ 229us) write_op.cc:270] Start()
0503 14:12:20.291075 (+ 25us) write_op.cc:276] Timestamp: P: 1714745532179928 usec, L: 0
0503 14:12:20.291213 (+ 138us) write_op.cc:270] Start()
0503 14:12:20.291237 (+ 24us) write_op.cc:276] Timestamp: P: 1714745532186554 usec, L: 0
0503 14:12:20.291371 (+ 134us) write_op.cc:270] Start()
0503 14:12:20.291390 (+ 19us) write_op.cc:276] Timestamp: P: 1714745532187284 usec, L: 0
0503 14:12:20.291522 (+ 132us) write_op.cc:270] Start()
0503 14:12:20.291543 (+ 21us) write_op.cc:276] Timestamp: P: 1714745532187961 usec, L: 0
0503 14:12:20.291675 (+ 132us) write_op.cc:270] Start()
0503 14:12:20.291698 (+ 23us) write_op.cc:276] Timestamp: P: 1714745532196616 usec, L: 0
0503 14:12:20.294037 (+ 2339us) write_op.cc:270] Start()
0503 14:12:20.294200 (+ 163us) write_op.cc:276] Timestamp: P: 1714745532200445 usec, L: 0
0503 14:12:20.295089 (+ 889us) write_op.cc:270] Start()
0503 14:12:20.295342 (+ 253us) write_op.cc:276] Timestamp: P: 1714745532204722 usec, L: 0
0503 14:12:20.296165 (+ 823us) write_op.cc:270] Start()
0503 14:12:20.296324 (+ 159us) write_op.cc:276] Timestamp: P: 1714745532206773 usec, L: 0
0503 14:12:20.296723 (+ 399us) write_op.cc:270] Start()
0503 14:12:20.296747 (+ 24us) write_op.cc:276] Timestamp: P: 1714745532240840 usec, L: 0
0503 14:12:20.297106 (+ 359us) write_op.cc:270] Start()
0503 14:12:20.297126 (+ 20us) write_op.cc:276] Timestamp: P: 1714745532287056 usec, L: 0
0503 14:12:20.297531 (+ 405us) write_op.cc:270] Start()
0503 14:12:20.297553 (+ 22us) write_op.cc:276] Timestamp: P: 1714745532288392 usec, L: 0
0503 14:12:20.297703 (+ 150us) write_op.cc:270] Start()
0503 14:12:20.297721 (+ 18us) write_op.cc:276] Timestamp: P: 1714745532289594 usec, L: 0
0503 14:12:20.297827 (+ 106us) write_op.cc:270] Start()
0503 14:12:20.297842 (+ 15us) write_op.cc:276] Timestamp: P: 1714745532290646 usec, L: 0
0503 14:12:20.298510 (+ 668us) write_op.cc:270] Start()
0503 14:12:20.298533 (+ 23us) write_op.cc:276] Timestamp: P: 1714745532291804 usec, L: 0
0503 14:12:20.298663 (+ 130us) write_op.cc:270] Start()
0503 14:12:20.298681 (+ 18us) write_op.cc:276] Timestamp: P: 1714745532296479 usec, L: 0
0503 14:12:20.298778 (+ 97us) write_op.cc:270] Start()
0503 14:12:20.298795 (+ 17us) write_op.cc:276] Timestamp: P: 1714745532308136 usec, L: 0
0503 14:12:20.298900 (+ 105us) write_op.cc:270] Start()
0503 14:12:20.298914 (+ 14us) write_op.cc:276] Timestamp: P: 1714745
I20240503 14:12:22.422394 28614 tablet_bootstrap.cc:492] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.422608 28614 ts_tablet_manager.cc:1367] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.010s
I20240503 14:12:22.424104 28614 raft_consensus.cc:348] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.424327 28614 raft_consensus.cc:374] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.424386 28614 raft_consensus.cc:729] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.424602 28614 consensus_queue.cc:260] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.425174 28614 ts_tablet_manager.cc:1398] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:22.425508 28614 tablet_bootstrap.cc:492] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.427680 28614 tablet_bootstrap.cc:654] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.428344 28653 ts_tablet_manager.cc:1888] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:22.428550 28653 log.cc:1196] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/9011de10a7ff497fb35ccb3ada7315ef
I20240503 14:12:22.429702 28600 tablet_bootstrap.cc:492] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:22.429932 28600 ts_tablet_manager.cc:1367] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.111s user 0.019s sys 0.000s
I20240503 14:12:22.430686 28600 raft_consensus.cc:348] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.430974 28600 raft_consensus.cc:374] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.431070 28600 raft_consensus.cc:729] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.431352 28600 consensus_queue.cc:260] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.432004 28600 ts_tablet_manager.cc:1398] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:22.432379 28600 tablet_bootstrap.cc:492] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.434453 28600 tablet_bootstrap.cc:654] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.422395 29828 raft_consensus.cc:2229] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.442890 29828 raft_consensus.cc:2258] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.443704 29828 ts_tablet_manager.cc:1875] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.448477 28653 ts_tablet_manager.cc:1909] T 9011de10a7ff497fb35ccb3ada7315ef P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.449671 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 760) took 2714 ms. Trace:
I20240503 14:12:22.449798 28653 rpcz_store.cc:276] 0503 14:12:19.735316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.735357 (+ 41us) service_pool.cc:225] Handling call
0503 14:12:22.374326 (+2638969us) ts_tablet_manager.cc:1120] Deleting tablet 9011de10a7ff497fb35ccb3ada7315ef
0503 14:12:22.374337 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.413237 (+ 38900us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.428214 (+ 14977us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.449634 (+ 21420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2638837}
I20240503 14:12:22.450449 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]) successfully deleted
I20240503 14:12:22.450584 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 783) took 1983 ms. Trace:
I20240503 14:12:22.450662 28653 rpcz_store.cc:276] 0503 14:12:20.466818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.466897 (+ 79us) service_pool.cc:225] Handling call
0503 14:12:22.450423 (+1983526us) ts_tablet_manager.cc:1120] Deleting tablet be238ee1d82548c9a1940e2a72e395c7
0503 14:12:22.450435 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.450562 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1983271}
W20240503 14:12:22.451120 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet be238ee1d82548c9a1940e2a72e395c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be238ee1d82548c9a1940e2a72e395c7 already in progress: creating tablet
I20240503 14:12:22.451179 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 784) took 1974 ms. Trace:
I20240503 14:12:22.451241 28653 rpcz_store.cc:276] 0503 14:12:20.477066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.477549 (+ 483us) service_pool.cc:225] Handling call
0503 14:12:22.451058 (+1973509us) ts_tablet_manager.cc:1120] Deleting tablet 332ecc478e49424d9e7d7decbbc2c29c
0503 14:12:22.451068 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.451152 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1973288}
I20240503 14:12:22.451700 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 785) took 1974 ms. Trace:
I20240503 14:12:22.451790 28653 rpcz_store.cc:276] 0503 14:12:20.477380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.477819 (+ 439us) service_pool.cc:225] Handling call
0503 14:12:22.451598 (+1973779us) ts_tablet_manager.cc:1120] Deleting tablet a31555fc800142bd849dbd572a194002
0503 14:12:22.451607 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.451682 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1973634}
W20240503 14:12:22.452214 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a31555fc800142bd849dbd572a194002 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a31555fc800142bd849dbd572a194002 already in progress: creating tablet
I20240503 14:12:22.461598 28614 tablet_bootstrap.cc:492] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.461797 28614 ts_tablet_manager.cc:1367] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.002s
I20240503 14:12:22.462612 28614 raft_consensus.cc:348] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.462826 28614 raft_consensus.cc:374] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.462884 28614 raft_consensus.cc:729] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.463121 28614 consensus_queue.cc:260] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.463673 28614 ts_tablet_manager.cc:1398] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:22.463987 28614 tablet_bootstrap.cc:492] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.465147 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 786) took 1987 ms. Trace:
I20240503 14:12:22.465279 28653 rpcz_store.cc:276] 0503 14:12:20.477476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.477998 (+ 522us) service_pool.cc:225] Handling call
0503 14:12:22.464934 (+1986936us) ts_tablet_manager.cc:1120] Deleting tablet 6615052ddbde487e840b6fd77a140e58
0503 14:12:22.464951 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.465109 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1985629}
W20240503 14:12:22.465864 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6615052ddbde487e840b6fd77a140e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6615052ddbde487e840b6fd77a140e58 already in progress: creating tablet
I20240503 14:12:22.467350 28614 tablet_bootstrap.cc:654] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.473346 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 788) took 1993 ms. Trace:
I20240503 14:12:22.473503 28653 rpcz_store.cc:276] 0503 14:12:20.479631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.479706 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:22.473127 (+1993421us) ts_tablet_manager.cc:1120] Deleting tablet b86c2df0470a4945b3e8207f2d965de2
0503 14:12:22.473144 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.473310 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1993245}
I20240503 14:12:22.475343 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 787) took 1995 ms. Trace:
I20240503 14:12:22.475462 28653 rpcz_store.cc:276] 0503 14:12:20.479492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.479964 (+ 472us) service_pool.cc:225] Handling call
0503 14:12:22.475175 (+1995211us) ts_tablet_manager.cc:1120] Deleting tablet f5bea5960cbb44d58d1dbabc888903c2
0503 14:12:22.475189 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.475318 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945040}
W20240503 14:12:22.477169 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b86c2df0470a4945b3e8207f2d965de2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b86c2df0470a4945b3e8207f2d965de2 already in progress: creating tablet
W20240503 14:12:22.477499 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f5bea5960cbb44d58d1dbabc888903c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5bea5960cbb44d58d1dbabc888903c2 already in progress: creating tablet
I20240503 14:12:22.478312 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 789) took 1947 ms. Trace:
I20240503 14:12:22.478421 28653 rpcz_store.cc:276] 0503 14:12:20.530631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.530680 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:22.478136 (+1947456us) ts_tablet_manager.cc:1120] Deleting tablet e708be06de744743aac4e192bdd0f449
0503 14:12:22.478151 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.478284 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945758}
I20240503 14:12:22.329756 29916 tablet.cc:1722] T 5d69cc969ea949a684a7e5aae237c13e P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:22.405201 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 74457us
I20240503 14:12:22.484164 29750 raft_consensus.cc:482] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.484390 29750 raft_consensus.cc:504] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.485105 29750 leader_election.cc:288] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:22.487069 28427 leader_election.cc:341] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.487543 28427 leader_election.cc:302] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.487380 29915 tablet.cc:1722] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:22.488631 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 790) took 1671 ms. Trace:
I20240503 14:12:22.488751 28653 rpcz_store.cc:276] 0503 14:12:20.817388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.817936 (+ 548us) service_pool.cc:225] Handling call
0503 14:12:22.488277 (+1670341us) ts_tablet_manager.cc:1120] Deleting tablet dcd405fd84d44076b1792b7fcbe00e84
0503 14:12:22.488385 (+ 108us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.488598 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655788}
W20240503 14:12:22.489338 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet dcd405fd84d44076b1792b7fcbe00e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcd405fd84d44076b1792b7fcbe00e84 already in progress: creating tablet
I20240503 14:12:22.489392 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 802) took 1604 ms. Trace:
I20240503 14:12:22.489462 28653 rpcz_store.cc:276] 0503 14:12:20.885027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.885095 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:22.489260 (+1604165us) ts_tablet_manager.cc:1120] Deleting tablet cf221f17ddfd461f83f7f916d7833011
0503 14:12:22.489271 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.489371 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1387007}
W20240503 14:12:22.492494 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet cf221f17ddfd461f83f7f916d7833011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf221f17ddfd461f83f7f916d7833011 already in progress: creating tablet
I20240503 14:12:22.492969 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 813) took 1403 ms. Trace:
I20240503 14:12:22.407136 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:22.497196 29864 raft_consensus.cc:482] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.497363 29864 raft_consensus.cc:504] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.498019 29864 leader_election.cc:288] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.498116 29701 raft_consensus.cc:2726] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:22.494069 28150 leader_election.cc:341] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.499262 28150 leader_election.cc:302] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:22.499719 29811 raft_consensus.cc:2726] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:22.499752 28427 leader_election.cc:341] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:22.499210 28653 rpcz_store.cc:276] 0503 14:12:21.089780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.187566 (+ 97786us) service_pool.cc:225] Handling call
0503 14:12:22.492798 (+1305232us) ts_tablet_manager.cc:1120] Deleting tablet a53279669bad40ad9243f4554a4e3c70
0503 14:12:22.492814 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.492944 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286589}
W20240503 14:12:22.500566 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a53279669bad40ad9243f4554a4e3c70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53279669bad40ad9243f4554a4e3c70 already in progress: creating tablet
W20240503 14:12:22.500617 28427 leader_election.cc:341] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.500617 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 814) took 1410 ms. Trace:
I20240503 14:12:22.501009 28653 rpcz_store.cc:276] 0503 14:12:21.089865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.193556 (+103691us) service_pool.cc:225] Handling call
0503 14:12:22.500464 (+1306908us) ts_tablet_manager.cc:1120] Deleting tablet 6f0d676fa1e74ab89922f7ba607fd2d8
0503 14:12:22.500477 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.500595 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290551}
W20240503 14:12:22.501470 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f0d676fa1e74ab89922f7ba607fd2d8 already in progress: creating tablet
I20240503 14:12:22.501894 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 812) took 1412 ms. Trace:
I20240503 14:12:22.501941 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:22.502327 28653 rpcz_store.cc:276] 0503 14:12:21.089678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.186928 (+ 97250us) service_pool.cc:225] Handling call
0503 14:12:22.501437 (+1314509us) ts_tablet_manager.cc:1120] Deleting tablet 4e11137b75794e2c9dd61ded326c868f
0503 14:12:22.501449 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.501876 (+ 427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278942}
W20240503 14:12:22.502750 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 4e11137b75794e2c9dd61ded326c868f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e11137b75794e2c9dd61ded326c868f already in progress: creating tablet
I20240503 14:12:22.502782 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:22.503124 28488 raft_consensus.cc:2454] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:22.503257 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 815) took 1413 ms. Trace:
I20240503 14:12:22.503760 28290 leader_election.cc:302] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:22.504132 29864 raft_consensus.cc:2781] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.504220 29864 raft_consensus.cc:482] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.504298 29864 raft_consensus.cc:3037] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.506903 29864 raft_consensus.cc:504] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.507652 29864 leader_election.cc:288] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.503938 28653 rpcz_store.cc:276] 0503 14:12:21.090006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.195294 (+105288us) service_pool.cc:225] Handling call
0503 14:12:22.503139 (+1307845us) ts_tablet_manager.cc:1120] Deleting tablet a3d1314841574fc8bcf3ccb16755e421
0503 14:12:22.503150 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.503240 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179743}
W20240503 14:12:22.508746 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a3d1314841574fc8bcf3ccb16755e421 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3d1314841574fc8bcf3ccb16755e421 already in progress: creating tablet
I20240503 14:12:22.509301 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 831) took 1202 ms. Trace:
I20240503 14:12:22.509874 28653 rpcz_store.cc:276] 0503 14:12:21.306576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.525702 (+219126us) service_pool.cc:225] Handling call
0503 14:12:22.508835 (+983133us) ts_tablet_manager.cc:1120] Deleting tablet b8d8f09130c04698949adc45f8b071d7
0503 14:12:22.508848 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.509274 (+ 426us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":893012}
W20240503 14:12:22.510807 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b8d8f09130c04698949adc45f8b071d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d8f09130c04698949adc45f8b071d7 already in progress: creating tablet
I20240503 14:12:22.511344 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 840) took 1065 ms. Trace:
I20240503 14:12:22.511780 28653 rpcz_store.cc:276] 0503 14:12:21.445617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.615852 (+170235us) service_pool.cc:225] Handling call
0503 14:12:22.510838 (+894986us) ts_tablet_manager.cc:1120] Deleting tablet 927d4359637f4ec189a5143e8201c7f1
0503 14:12:22.510849 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.511320 (+ 471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":894818}
I20240503 14:12:22.512253 29864 raft_consensus.cc:482] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.512396 29864 raft_consensus.cc:504] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:22.513038 29864 leader_election.cc:288] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:22.513795 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 927d4359637f4ec189a5143e8201c7f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 927d4359637f4ec189a5143e8201c7f1 already in progress: creating tablet
I20240503 14:12:22.514340 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 841) took 1068 ms. Trace:
I20240503 14:12:22.514694 28653 rpcz_store.cc:276] 0503 14:12:21.445783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.616033 (+170250us) service_pool.cc:225] Handling call
0503 14:12:22.513780 (+897747us) ts_tablet_manager.cc:1120] Deleting tablet e8f9f10348014293a421ed11a50f89e7
0503 14:12:22.513792 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.514314 (+ 522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":897577}
W20240503 14:12:22.515579 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e8f9f10348014293a421ed11a50f89e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8f9f10348014293a421ed11a50f89e7 already in progress: creating tablet
I20240503 14:12:22.515974 28653 tablet_replica.cc:330] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.516394 28653 raft_consensus.cc:2229] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.517095 28653 raft_consensus.cc:2258] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.518148 28653 ts_tablet_manager.cc:1875] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.519101 29915 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51314 (request call id 866) took 1119 ms. Trace:
I20240503 14:12:22.519260 29915 rpcz_store.cc:276] 0503 14:12:21.399305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.399543 (+ 238us) service_pool.cc:225] Handling call
0503 14:12:22.519073 (+1119530us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:21.399991 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0503 14:12:21.400120 (+ 129us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0503 14:12:21.400129 (+ 9us) alter_schema_op.cc:59] Acquired schema lock
0503 14:12:21.400135 (+ 6us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0503 14:12:21.400191 (+ 56us) alter_schema_op.cc:119] START. Timestamp: 1714745541398561
0503 14:12:21.400196 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:21.400416 (+ 220us) log.cc:842] Serialized 173 byte log entry
0503 14:12:22.485697 (+1085281us) op_driver.cc:444] REPLICATION: finished
0503 14:12:22.487355 (+ 1658us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0503 14:12:22.488402 (+ 1047us) log.cc:842] Serialized 5 byte log entry
0503 14:12:22.518399 (+ 29997us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.518763 (+ 364us) log.cc:842] Serialized 14 byte log entry
0503 14:12:22.519026 (+ 263us) alter_schema_op.cc:65] Released schema lock
0503 14:12:22.519031 (+ 5us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1571,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":6248,"prepare.run_wall_time_us":107719,"raft.queue_time_us":73,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":1085468,"thread_start_us":104504,"threads_started":1,"wal-append.queue_time_us":103943}]]}
I20240503 14:12:22.520735 29811 raft_consensus.cc:482] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.520886 29811 raft_consensus.cc:504] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.521566 29811 leader_election.cc:288] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.523308 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:22.523936 28150 leader_election.cc:341] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.526726 29774 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:22.527161 29774 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:22.537604 28653 ts_tablet_manager.cc:1888] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:22.504228 28290 leader_election.cc:341] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.541435 28653 log.cc:1196] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/8410e8b6b511403ea609b1353855937b
I20240503 14:12:22.539645 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:22.542304 28653 ts_tablet_manager.cc:1909] T 8410e8b6b511403ea609b1353855937b P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:22.543251 28290 leader_election.cc:341] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 65cdd090bf0f493f8b92341369378532
W20240503 14:12:22.543493 28290 leader_election.cc:341] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 65cdd090bf0f493f8b92341369378532
I20240503 14:12:22.543582 28290 leader_election.cc:302] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:22.544201 29864 raft_consensus.cc:2726] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.544654 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 842) took 1098 ms. Trace:
W20240503 14:12:22.544900 28290 leader_election.cc:341] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.546535 28653 rpcz_store.cc:276] 0503 14:12:21.445954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.616218 (+170264us) service_pool.cc:225] Handling call
0503 14:12:22.515641 (+899423us) ts_tablet_manager.cc:1120] Deleting tablet 8410e8b6b511403ea609b1353855937b
0503 14:12:22.515653 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.526303 (+ 10650us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.537474 (+ 11171us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.544634 (+ 7160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":899291}
I20240503 14:12:22.547814 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 843) took 1101 ms. Trace:
I20240503 14:12:22.548631 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 8410e8b6b511403ea609b1353855937b (table default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280 [id=05881d1136974066b34909e625c970a7]) successfully deleted
I20240503 14:12:22.548661 28653 rpcz_store.cc:276] 0503 14:12:21.446057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.616354 (+170297us) service_pool.cc:225] Handling call
0503 14:12:22.547657 (+931303us) ts_tablet_manager.cc:1120] Deleting tablet 3112560e16f14b4e89aae90c014be9fb
0503 14:12:22.547670 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.547796 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":931202}
W20240503 14:12:22.549753 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3112560e16f14b4e89aae90c014be9fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3112560e16f14b4e89aae90c014be9fb already in progress: creating tablet
I20240503 14:12:22.550174 28653 tablet_replica.cc:330] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.550606 28653 raft_consensus.cc:2229] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.551297 28653 raft_consensus.cc:2258] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.552310 28653 ts_tablet_manager.cc:1875] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.559713 28653 ts_tablet_manager.cc:1888] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.569147 29811 raft_consensus.cc:482] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.572912 29811 raft_consensus.cc:504] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.573696 29811 leader_election.cc:288] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.574481 28653 log.cc:1196] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/5a081ba59ad44890bf2f94874c034277
I20240503 14:12:22.575177 28653 ts_tablet_manager.cc:1909] T 5a081ba59ad44890bf2f94874c034277 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.575114 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d28d3c2ea472191ad21355248a2fc" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:22.575632 28150 leader_election.cc:341] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.577596 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 832) took 1263 ms. Trace:
I20240503 14:12:22.578037 29867 raft_consensus.cc:504] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:22.579557 28653 rpcz_store.cc:276] 0503 14:12:21.314364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.527537 (+213173us) service_pool.cc:225] Handling call
0503 14:12:22.549833 (+1022296us) ts_tablet_manager.cc:1120] Deleting tablet 5a081ba59ad44890bf2f94874c034277
0503 14:12:22.549845 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.556777 (+ 6932us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.559608 (+ 2831us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.577571 (+ 17963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933157}
I20240503 14:12:22.580649 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 5a081ba59ad44890bf2f94874c034277 (table default.Table_1145b045780c4d1e8fedd333eea661d4 [id=114dfd39b4ea4958a08161a34e4a1f9f]) successfully deleted
I20240503 14:12:22.581163 28653 tablet_replica.cc:330] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.581629 28653 raft_consensus.cc:2229] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.582409 28653 raft_consensus.cc:2258] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.584316 28653 ts_tablet_manager.cc:1875] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.539240 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:22.589133 28486 raft_consensus.cc:3037] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:22.560714 28427 leader_election.cc:341] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 2f89fca0480d4bf28160384ae0421bce
I20240503 14:12:22.574582 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d28d3c2ea472191ad21355248a2fc" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:22.589859 28604 tablet_bootstrap.cc:492] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:22.590008 28604 ts_tablet_manager.cc:1367] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.297s user 0.043s sys 0.066s
W20240503 14:12:22.590534 28150 leader_election.cc:341] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.590675 28150 leader_election.cc:302] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:22.590710 28427 leader_election.cc:341] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:22.590998 28427 leader_election.cc:341] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.591086 28427 leader_election.cc:302] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.591123 29811 raft_consensus.cc:2726] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:22.591454 28427 leader_election.cc:341] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 9011de10a7ff497fb35ccb3ada7315ef
W20240503 14:12:22.591910 28427 leader_election.cc:341] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.592497 28427 leader_election.cc:302] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.592859 28486 raft_consensus.cc:2454] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
W20240503 14:12:22.593338 28427 leader_election.cc:341] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.593536 28427 leader_election.cc:302] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.547343 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:22.593899 28488 raft_consensus.cc:2454] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:22.592804 28604 raft_consensus.cc:348] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.594343 28150 leader_election.cc:302] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.594797 29811 raft_consensus.cc:2781] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.594872 29811 raft_consensus.cc:482] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.594925 29811 raft_consensus.cc:3037] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.597530 28614 tablet_bootstrap.cc:492] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.597718 28614 ts_tablet_manager.cc:1367] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.134s user 0.012s sys 0.016s
I20240503 14:12:22.597728 29811 raft_consensus.cc:504] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.598446 29811 leader_election.cc:288] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.598544 28614 raft_consensus.cc:348] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.598778 28614 raft_consensus.cc:374] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.598830 28614 raft_consensus.cc:729] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.599051 28614 consensus_queue.cc:260] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.599202 28604 raft_consensus.cc:374] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.599601 28604 raft_consensus.cc:729] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.599653 28614 ts_tablet_manager.cc:1398] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:22.599979 28614 tablet_bootstrap.cc:492] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.599997 28604 consensus_queue.cc:260] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.601718 28614 tablet_bootstrap.cc:654] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.603021 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:22.603677 28150 leader_election.cc:341] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.603845 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:22.604146 28488 raft_consensus.cc:3037] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:22.606670 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0c72d1569de4810853eebb533d77c9b consensus state has no leader
I20240503 14:12:22.607139 28653 ts_tablet_manager.cc:1888] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.609908 28653 log.cc:1196] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:22.610519 28653 ts_tablet_manager.cc:1909] T e0a92fc3833e49aaab8c5415de9bb685 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.611136 28488 raft_consensus.cc:2454] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:22.611675 28150 leader_election.cc:302] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.612033 28614 tablet_bootstrap.cc:492] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.612185 29811 raft_consensus.cc:2781] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.612228 28614 ts_tablet_manager.cc:1367] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:12:22.612305 29811 raft_consensus.cc:686] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:22.612552 29811 consensus_queue.cc:237] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.613020 28614 raft_consensus.cc:348] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.613253 28614 raft_consensus.cc:374] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.613302 28614 raft_consensus.cc:729] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.613530 28614 consensus_queue.cc:260] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.614398 28614 ts_tablet_manager.cc:1398] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:22.614713 28614 tablet_bootstrap.cc:492] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.616446 28614 tablet_bootstrap.cc:654] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.619529 28290 leader_election.cc:302] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:22.620199 29864 raft_consensus.cc:2781] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.621063 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 833) took 1301 ms. Trace:
I20240503 14:12:22.621804 28653 rpcz_store.cc:276] 0503 14:12:21.319530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.532958 (+213428us) service_pool.cc:225] Handling call
0503 14:12:22.580758 (+1047800us) ts_tablet_manager.cc:1120] Deleting tablet e0a92fc3833e49aaab8c5415de9bb685
0503 14:12:22.580778 (+ 20us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.601742 (+ 20964us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.607004 (+ 5262us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.621031 (+ 14027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963951}
I20240503 14:12:22.622756 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e0a92fc3833e49aaab8c5415de9bb685 (table default.Table_9953322d44f34ce5ad1123ba2784b66f [id=5ae48705fb644bddba8e69d66c640a68]) successfully deleted
I20240503 14:12:22.622838 28653 tablet_replica.cc:330] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.623433 28653 raft_consensus.cc:2229] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.623984 28653 raft_consensus.cc:2258] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.625003 28653 ts_tablet_manager.cc:1875] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.625829 28614 tablet_bootstrap.cc:492] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.626026 28614 ts_tablet_manager.cc:1367] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 14:12:22.627082 28614 raft_consensus.cc:348] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.627362 28614 raft_consensus.cc:374] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.627421 28614 raft_consensus.cc:729] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.627678 28614 consensus_queue.cc:260] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.631207 28032 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]). Expected version 3 got 0
I20240503 14:12:22.632396 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d201fddd164546b5a2d529d6bc79cb36" } new_table_name: "default.Table_8eb9b1e15db5432e8c3398239f7584b1" modify_external_catalogs: true
I20240503 14:12:22.635097 28653 ts_tablet_manager.cc:1888] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.635259 28653 log.cc:1196] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/95c97a43bd714da68c1a77408f3d1cd1
I20240503 14:12:22.635654 28653 ts_tablet_manager.cc:1909] T 95c97a43bd714da68c1a77408f3d1cd1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.636689 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 835) took 1204 ms. Trace:
I20240503 14:12:22.636884 28653 rpcz_store.cc:276] 0503 14:12:21.431916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.537669 (+105753us) service_pool.cc:225] Handling call
0503 14:12:22.622759 (+1085090us) ts_tablet_manager.cc:1120] Deleting tablet 95c97a43bd714da68c1a77408f3d1cd1
0503 14:12:22.622783 (+ 24us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.631757 (+ 8974us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.634953 (+ 3196us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.636664 (+ 1711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005881}
I20240503 14:12:22.637856 28653 tablet_replica.cc:330] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.637842 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 95c97a43bd714da68c1a77408f3d1cd1 (table default.Table_c7035b3f783948498dfb7bd5ed14aad8 [id=15ae3079abdd4069aaeb84e1d85da988]) successfully deleted
I20240503 14:12:22.638185 28653 raft_consensus.cc:2229] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.638433 28653 raft_consensus.cc:2258] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.639148 28653 ts_tablet_manager.cc:1875] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.640573 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_10dee84bef6148bf8704fd1da1e39b58" } modify_external_catalogs: true
I20240503 14:12:22.640893 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_10dee84bef6148bf8704fd1da1e39b58" } modify_external_catalogs: true
I20240503 14:12:22.641018 28614 ts_tablet_manager.cc:1398] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20240503 14:12:22.641410 28614 tablet_bootstrap.cc:492] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.641947 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6aea0eb4f439481da1da149c63f49de8" } new_table_name: "default.Table_3a027411e7bc4b75a89b58229859f07e" modify_external_catalogs: true
I20240503 14:12:22.643646 28614 tablet_bootstrap.cc:654] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.647816 28653 ts_tablet_manager.cc:1888] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.648072 28653 log.cc:1196] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/1c5e3649c71f4d4da710c69f85a82c25
I20240503 14:12:22.648514 28653 ts_tablet_manager.cc:1909] T 1c5e3649c71f4d4da710c69f85a82c25 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.649533 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 836) took 1204 ms. Trace:
I20240503 14:12:22.649789 28653 rpcz_store.cc:276] 0503 14:12:21.445130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.539616 (+ 94486us) service_pool.cc:225] Handling call
0503 14:12:22.637790 (+1098174us) ts_tablet_manager.cc:1120] Deleting tablet 1c5e3649c71f4d4da710c69f85a82c25
0503 14:12:22.637803 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.643964 (+ 6161us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.647674 (+ 3710us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.649503 (+ 1829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020839}
I20240503 14:12:22.650506 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 1c5e3649c71f4d4da710c69f85a82c25 (table default.Table_dc507e002d97466bae378a75112560a8 [id=3a0db192d94f40c3b69f089100ad18ae]) successfully deleted
I20240503 14:12:22.650635 29701 raft_consensus.cc:2726] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.650849 28653 tablet_replica.cc:330] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.650898 29701 raft_consensus.cc:2726] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.651113 29701 raft_consensus.cc:2726] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.651233 28653 raft_consensus.cc:2229] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:22.651567 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 58240us
I20240503 14:12:22.651794 28604 ts_tablet_manager.cc:1398] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
W20240503 14:12:22.651876 28427 leader_election.cc:341] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:22.652151 28604 tablet_bootstrap.cc:492] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:22.653210 29922 tablet.cc:1722] T 65853212c2404597b015ddc8ea99ee47 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:22.654004 28604 tablet_bootstrap.cc:654] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.654901 29867 leader_election.cc:288] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:22.657317 29901 raft_consensus.cc:482] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.657491 29901 raft_consensus.cc:504] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.658094 29901 leader_election.cc:288] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.659147 28614 tablet_bootstrap.cc:492] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.659333 28614 ts_tablet_manager.cc:1367] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20240503 14:12:22.652058 28427 leader_election.cc:302] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:22.659507 28427 raft_consensus.cc:2670] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:22.659715 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd8b2adaeeb4c3abfbfe68cc005e03a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:22.660224 28614 raft_consensus.cc:348] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.660451 28614 raft_consensus.cc:374] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.660506 28614 raft_consensus.cc:729] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.660727 28614 consensus_queue.cc:260] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.661828 28614 ts_tablet_manager.cc:1398] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:22.662176 28614 tablet_bootstrap.cc:492] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.664783 28614 tablet_bootstrap.cc:654] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.666478 29864 raft_consensus.cc:686] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:22.666968 29864 consensus_queue.cc:237] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.667973 29921 tablet.cc:1722] T 65853212c2404597b015ddc8ea99ee47 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:22.669409 28290 leader_election.cc:341] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 5dd8b2adaeeb4c3abfbfe68cc005e03a
I20240503 14:12:22.669756 28290 leader_election.cc:302] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:22.671979 29867 raft_consensus.cc:2726] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.674755 28661 ts_tablet_manager.cc:1888] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.679910 28614 tablet_bootstrap.cc:492] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.680025 28653 raft_consensus.cc:2258] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.680120 28614 ts_tablet_manager.cc:1367] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240503 14:12:22.680827 28653 ts_tablet_manager.cc:1875] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.681022 28614 raft_consensus.cc:348] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.681280 28614 raft_consensus.cc:374] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.681342 28614 raft_consensus.cc:729] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.681561 28614 consensus_queue.cc:260] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.682179 28614 ts_tablet_manager.cc:1398] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:22.682516 28614 tablet_bootstrap.cc:492] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.684693 28614 tablet_bootstrap.cc:654] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.693308 28614 tablet_bootstrap.cc:492] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.693508 28614 ts_tablet_manager.cc:1367] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240503 14:12:22.694931 28614 raft_consensus.cc:348] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.695194 28614 raft_consensus.cc:374] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.695250 28614 raft_consensus.cc:729] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.695466 28614 consensus_queue.cc:260] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.696010 28614 ts_tablet_manager.cc:1398] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:22.699049 28653 ts_tablet_manager.cc:1888] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.699268 28653 log.cc:1196] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7b6c998e518541248a87e4c1551ec080
I20240503 14:12:22.699755 28653 ts_tablet_manager.cc:1909] T 7b6c998e518541248a87e4c1551ec080 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.700685 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 834) took 1369 ms. Trace:
I20240503 14:12:22.700846 28653 rpcz_store.cc:276] 0503 14:12:21.331620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.534381 (+202761us) service_pool.cc:225] Handling call
0503 14:12:22.650622 (+1116241us) ts_tablet_manager.cc:1120] Deleting tablet 7b6c998e518541248a87e4c1551ec080
0503 14:12:22.650723 (+ 101us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.695775 (+ 45052us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.698928 (+ 3153us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.700656 (+ 1728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1033595}
I20240503 14:12:22.705185 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 839) took 1259 ms. Trace:
I20240503 14:12:22.705132 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7b6c998e518541248a87e4c1551ec080 (table default.Table_a7aefabd0c3841618e68382b8fa41a9a [id=c23a0b9e98f949559497aea2a4dab24a]) successfully deleted
I20240503 14:12:22.705421 28653 rpcz_store.cc:276] 0503 14:12:21.445499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.552938 (+107439us) service_pool.cc:225] Handling call
0503 14:12:22.705023 (+1152085us) ts_tablet_manager.cc:1120] Deleting tablet 8097ba5606cf42d68448833de36d85c7
0503 14:12:22.705035 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.705163 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087888}
I20240503 14:12:22.705766 29828 ts_tablet_manager.cc:1888] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:22.705979 29828 log.cc:1196] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:22.706149 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 838) took 1260 ms. Trace:
I20240503 14:12:22.706378 29828 ts_tablet_manager.cc:1909] T 0af0342e0e0b47499a9322b5b2cb3d03 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:22.706269 28653 rpcz_store.cc:276] 0503 14:12:21.445400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.545595 (+100195us) service_pool.cc:225] Handling call
0503 14:12:22.705987 (+1160392us) ts_tablet_manager.cc:1120] Deleting tablet 37dac592c9dd4888a2ce89fa922f7fc8
0503 14:12:22.705999 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.706130 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074122}
W20240503 14:12:22.706722 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 37dac592c9dd4888a2ce89fa922f7fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37dac592c9dd4888a2ce89fa922f7fc8 already in progress: creating tablet
I20240503 14:12:22.706764 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 849) took 1045 ms. Trace:
I20240503 14:12:22.706842 28653 rpcz_store.cc:276] 0503 14:12:21.661775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.664562 (+ 2787us) service_pool.cc:225] Handling call
0503 14:12:22.706648 (+1042086us) ts_tablet_manager.cc:1120] Deleting tablet 10876f4512794e6aa72bda3c1da8b2d0
0503 14:12:22.706658 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.706747 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041446}
I20240503 14:12:22.706830 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 863) took 1152 ms. Trace:
I20240503 14:12:22.707018 29828 rpcz_store.cc:276] 0503 14:12:21.554367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.560359 (+ 5992us) service_pool.cc:225] Handling call
0503 14:12:22.387494 (+827135us) ts_tablet_manager.cc:1120] Deleting tablet 0af0342e0e0b47499a9322b5b2cb3d03
0503 14:12:22.387505 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.648626 (+261121us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.705503 (+ 56877us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.706796 (+ 1293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":826859}
W20240503 14:12:22.707342 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 10876f4512794e6aa72bda3c1da8b2d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10876f4512794e6aa72bda3c1da8b2d0 already in progress: creating tablet
I20240503 14:12:22.707403 29828 tablet_replica.cc:330] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
W20240503 14:12:22.707818 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 752b369f24224f2486834814a9d08495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 752b369f24224f2486834814a9d08495 already in progress: creating tablet
W20240503 14:12:22.708107 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 281667db9a9c43d3b9d7dd9ee14b4b06 already in progress: creating tablet
I20240503 14:12:22.708230 28614 tablet_bootstrap.cc:492] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:22.708412 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d952b436198141ad80fef5fb1ea1e736 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d952b436198141ad80fef5fb1ea1e736 already in progress: creating tablet
W20240503 14:12:22.708703 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 60d4f5591b1f4826bc09e57094fb46bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d4f5591b1f4826bc09e57094fb46bf already in progress: creating tablet
I20240503 14:12:22.708930 28653 tablet_replica.cc:330] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.709123 28653 raft_consensus.cc:2229] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.709342 28653 raft_consensus.cc:2258] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.710278 28614 tablet_bootstrap.cc:654] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.710067 28653 ts_tablet_manager.cc:1875] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.713830 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet b25c852aa7aa4fbfafdacec43f5bc426 (table default.Table_3e4ec426791c4b11b43d3c36260eedd4 [id=c14f7d47ddd64652aa553945f775aec6]) successfully deleted
I20240503 14:12:22.714679 28217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:22.715672 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4dc732bcb44222b4b15e5d77defe33" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:22.716213 28206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bae128469f4e8bb2d743320731446a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:22.716400 28206 raft_consensus.cc:3037] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.717547 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0af0342e0e0b47499a9322b5b2cb3d03 (table default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]) successfully deleted
I20240503 14:12:22.718643 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4dc732bcb44222b4b15e5d77defe33" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:22.718906 28353 raft_consensus.cc:2454] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:22.719089 28206 raft_consensus.cc:2454] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:22.719594 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bae128469f4e8bb2d743320731446a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:22.719761 28353 raft_consensus.cc:3037] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.722836 28353 raft_consensus.cc:2454] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:22.724439 29921 tablet.cc:1722] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:22.726116 28614 tablet_bootstrap.cc:492] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.726289 28614 ts_tablet_manager.cc:1367] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20240503 14:12:22.727370 28614 raft_consensus.cc:348] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.727617 28614 raft_consensus.cc:374] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.727675 28614 raft_consensus.cc:729] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.727892 28614 consensus_queue.cc:260] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.728602 28614 ts_tablet_manager.cc:1398] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:22.728950 28614 tablet_bootstrap.cc:492] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.732388 28614 tablet_bootstrap.cc:654] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.734443 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:22.734966 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 02dd4c914a4846bab3a00408247ec64a (table default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc]). Expected version 2 got 0
I20240503 14:12:22.739727 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 238b26ebe543422ebef15b0d2d1302df (DEFAULT_TABLE table=default.Table_39fec5701a7a485caa43e8694f08bbf9 [id=8c00d81692cc446a943f33f8f80d4290]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.739851 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet ed4dc732bcb44222b4b15e5d77defe33 (table default.Table_3ce7004b7c164629a92e6bcbb421348d [id=a6a9ad0f171448c2a3d3401e02ec62c0]). Expected version 4 got 0
I20240503 14:12:22.740213 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238b26ebe543422ebef15b0d2d1302df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.745018 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 5500da0e5658492b840dd29273ab265f (DEFAULT_TABLE table=default.Table_70df8ea2254f45aab971123dd7f55a44 [id=24611f8249494aa094924164a4f2c1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.745213 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 95a74f927c974e8aa72ba1462f78116c (DEFAULT_TABLE table=default.Table_664e113b7fe34816ad9b995b341c40ef [id=376eb95d389d4731aca8acc0b1617a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.745433 28180 tablet_service.cc:1467] Processing CreateTablet for tablet ceec6992b0474706b8b3766518b0c274 (DEFAULT_TABLE table=default.Table_decc999a1649405b8e7301584192c536 [id=ccb4698582a54cdcac8066d50c7af7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.745047 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda92c8e3c94139b84eabc81127613e (DEFAULT_TABLE table=default.Table_463c966bd4c847d0a0967bc7dae04ec6 [id=5c1c5cfd2cef4bd6ad984e404026d90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.745448 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5500da0e5658492b840dd29273ab265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.746343 28186 tablet_service.cc:1467] Processing CreateTablet for tablet a526e9f1e9934e519349dcd4523ee694 (DEFAULT_TABLE table=default.Table_55224d5639254fa1a4890def9934e40a [id=84b9cf0bbc494213a97d9f1aa7ce30ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.746466 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a74f927c974e8aa72ba1462f78116c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.746670 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a526e9f1e9934e519349dcd4523ee694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.746896 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda92c8e3c94139b84eabc81127613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.747037 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceec6992b0474706b8b3766518b0c274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.756780 28192 tablet_service.cc:1467] Processing CreateTablet for tablet bf9e98bee96d4df895eaed639d033b21 (DEFAULT_TABLE table=default.Table_7e8c0ee351544a32a03b6aa78c2f9264 [id=20cebd5f519640beac7b6ac35e3be8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.757203 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9e98bee96d4df895eaed639d033b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.758410 28195 tablet_service.cc:1467] Processing CreateTablet for tablet abb9279e91124fe6bfbdc8856eccfed8 (DEFAULT_TABLE table=default.Table_c1936a618b9f488fbd38c45bbc95303c [id=1654ff04429e4f1397b299342743bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.758868 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb9279e91124fe6bfbdc8856eccfed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.763541 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac145bcc44b4437b840df389cd254f9 (DEFAULT_TABLE table=default.Table_e5c58148b8e649f482d8025f0d340f3b [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.764021 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac145bcc44b4437b840df389cd254f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.767786 28179 tablet_service.cc:1467] Processing CreateTablet for tablet ffe63f0b6e1341089f465f3e37da419c (DEFAULT_TABLE table=default.Table_0d6cc797740e4f18a7aad22805859cd5 [id=1a3c8687e5d84377a3215cb23362f78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.768246 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe63f0b6e1341089f465f3e37da419c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.768869 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3ae54bb8b34f4983e32a5ae664dd66 (DEFAULT_TABLE table=default.Table_c6d80a93009d467ab3bc0bc3c3ba529f [id=13124c075b8841d6a27878706e9c9ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.769409 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3ae54bb8b34f4983e32a5ae664dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.772720 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:22.772845 28186 tablet_service.cc:1467] Processing CreateTablet for tablet ea5221fc8d6548e08e4ae2c2d466f970 (DEFAULT_TABLE table=default.Table_04ae17bd98c04c4b993d808882cdc3ec [id=5629316959054b54a63bac11c6d3d8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.773195 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5221fc8d6548e08e4ae2c2d466f970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.774969 28614 tablet_bootstrap.cc:492] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.775133 28614 ts_tablet_manager.cc:1367] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20240503 14:12:22.776015 28614 raft_consensus.cc:348] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.776268 28614 raft_consensus.cc:374] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.776319 28614 raft_consensus.cc:729] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.776619 28193 tablet_service.cc:1467] Processing CreateTablet for tablet a7cbb1ecc2294e27a41b61693ac21e9a (DEFAULT_TABLE table=default.Table_f526ebddbfd04608bdf15d7af168e12a [id=81571ea109384d17b612344b854e47dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.776772 28614 consensus_queue.cc:260] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.777366 28614 ts_tablet_manager.cc:1398] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:22.777675 28614 tablet_bootstrap.cc:492] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.777949 29922 tablet.cc:1722] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:22.779711 28192 tablet_service.cc:1467] Processing CreateTablet for tablet ed8cda887ec7497c9950c182e5c2852b (DEFAULT_TABLE table=default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.780139 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8cda887ec7497c9950c182e5c2852b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.780635 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cbb1ecc2294e27a41b61693ac21e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.677062 29915 tablet.cc:1722] T 65853212c2404597b015ddc8ea99ee47 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:22.697324 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet dda39f0323ec47799d3d44c96f3ba605 with delete_type TABLET_DATA_DELETED (replaced by tablet e750765798db487fb0b5142f01d21771) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:22.697695 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 41cbd1326869434594e126421ad132d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:22.708231 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet b04fff656f624001a709f783816eec1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:22.711192 28541 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:22.790199 28614 tablet_bootstrap.cc:654] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.792876 28661 log.cc:1196] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/5775006e9f8f4612aa70dfb316de9ea1
I20240503 14:12:22.793316 28661 ts_tablet_manager.cc:1909] T 5775006e9f8f4612aa70dfb316de9ea1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:22.794178 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 738) took 4210 ms. Trace:
I20240503 14:12:22.794327 28661 rpcz_store.cc:276] 0503 14:12:18.583321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.583376 (+ 55us) service_pool.cc:225] Handling call
0503 14:12:22.314810 (+3731434us) ts_tablet_manager.cc:1120] Deleting tablet 5775006e9f8f4612aa70dfb316de9ea1
0503 14:12:22.314821 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.595693 (+280872us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.674629 (+ 78936us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.794152 (+119523us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12468,"tablet-delete.queue_time_us":3731215}
I20240503 14:12:22.794768 28661 tablet_replica.cc:330] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:22.795007 28661 raft_consensus.cc:2229] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.795215 28661 raft_consensus.cc:2258] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.795874 28661 ts_tablet_manager.cc:1875] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.798202 28614 tablet_bootstrap.cc:492] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.798369 28614 ts_tablet_manager.cc:1367] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20240503 14:12:22.799500 28614 raft_consensus.cc:348] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.799711 28614 raft_consensus.cc:374] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.799772 28614 raft_consensus.cc:729] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.799981 28614 consensus_queue.cc:260] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.800521 28614 ts_tablet_manager.cc:1398] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:22.800819 28614 tablet_bootstrap.cc:492] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.800814 28600 tablet_bootstrap.cc:492] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:22.800958 28600 ts_tablet_manager.cc:1367] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.369s user 0.138s sys 0.151s
I20240503 14:12:22.801565 28600 raft_consensus.cc:348] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.801746 28600 raft_consensus.cc:374] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.801798 28600 raft_consensus.cc:729] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.802084 28604 tablet_bootstrap.cc:492] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:22.801987 28600 consensus_queue.cc:260] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.802567 28600 ts_tablet_manager.cc:1398] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:22.802989 28600 tablet_bootstrap.cc:492] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.804193 28604 ts_tablet_manager.cc:1367] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.152s user 0.022s sys 0.000s
I20240503 14:12:22.804848 28600 tablet_bootstrap.cc:654] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.804971 28614 tablet_bootstrap.cc:654] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.804980 28604 raft_consensus.cc:348] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.805258 28604 raft_consensus.cc:374] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.805363 28604 raft_consensus.cc:729] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.805608 28604 consensus_queue.cc:260] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.808166 28604 ts_tablet_manager.cc:1398] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:22.808555 28604 tablet_bootstrap.cc:492] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:22.809928 28604 tablet_bootstrap.cc:654] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.812811 28600 tablet_bootstrap.cc:492] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:22.812933 28600 ts_tablet_manager.cc:1367] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 14:12:22.813732 28600 raft_consensus.cc:348] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.813905 28600 raft_consensus.cc:374] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.813964 28600 raft_consensus.cc:729] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.814211 28600 consensus_queue.cc:260] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.814798 28600 ts_tablet_manager.cc:1398] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:22.815102 28600 tablet_bootstrap.cc:492] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.817879 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 5775006e9f8f4612aa70dfb316de9ea1 (table default.Table_b052ceeaac3644e18e8f3cb1947a3863 [id=a61553d45d10436eb38fd35c53417842]) successfully deleted
I20240503 14:12:22.818395 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 238b26ebe543422ebef15b0d2d1302df (DEFAULT_TABLE table=default.Table_39fec5701a7a485caa43e8694f08bbf9 [id=8c00d81692cc446a943f33f8f80d4290]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.818799 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238b26ebe543422ebef15b0d2d1302df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.822165 28600 tablet_bootstrap.cc:654] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.823524 28614 tablet_bootstrap.cc:492] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.823662 28614 ts_tablet_manager.cc:1367] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 14:12:22.824605 28614 raft_consensus.cc:348] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.824822 28614 raft_consensus.cc:374] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.824867 28614 raft_consensus.cc:729] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.825078 28614 consensus_queue.cc:260] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.825335 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 238b26ebe543422ebef15b0d2d1302df (DEFAULT_TABLE table=default.Table_39fec5701a7a485caa43e8694f08bbf9 [id=8c00d81692cc446a943f33f8f80d4290]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.825634 28614 ts_tablet_manager.cc:1398] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:22.825713 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238b26ebe543422ebef15b0d2d1302df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.825902 28614 tablet_bootstrap.cc:492] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.827996 28614 tablet_bootstrap.cc:654] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.830273 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 5500da0e5658492b840dd29273ab265f (DEFAULT_TABLE table=default.Table_70df8ea2254f45aab971123dd7f55a44 [id=24611f8249494aa094924164a4f2c1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.830753 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5500da0e5658492b840dd29273ab265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.842031 29903 raft_consensus.cc:482] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.842229 29903 raft_consensus.cc:504] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.842901 29903 leader_election.cc:288] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.844494 28461 tablet_service.cc:1467] Processing CreateTablet for tablet ceec6992b0474706b8b3766518b0c274 (DEFAULT_TABLE table=default.Table_decc999a1649405b8e7301584192c536 [id=ccb4698582a54cdcac8066d50c7af7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.844676 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda92c8e3c94139b84eabc81127613e (DEFAULT_TABLE table=default.Table_463c966bd4c847d0a0967bc7dae04ec6 [id=5c1c5cfd2cef4bd6ad984e404026d90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.844940 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceec6992b0474706b8b3766518b0c274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.845569 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:22.845784 28493 raft_consensus.cc:2379] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:22.844494 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 95a74f927c974e8aa72ba1462f78116c (DEFAULT_TABLE table=default.Table_664e113b7fe34816ad9b995b341c40ef [id=376eb95d389d4731aca8acc0b1617a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.846830 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a74f927c974e8aa72ba1462f78116c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.847448 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda92c8e3c94139b84eabc81127613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.855556 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221218847554cbd9d3242a9a6431e46" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:22.856385 28150 leader_election.cc:302] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:22.856819 29903 raft_consensus.cc:3037] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.858319 28462 tablet_service.cc:1467] Processing CreateTablet for tablet a526e9f1e9934e519349dcd4523ee694 (DEFAULT_TABLE table=default.Table_55224d5639254fa1a4890def9934e40a [id=84b9cf0bbc494213a97d9f1aa7ce30ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.858850 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a526e9f1e9934e519349dcd4523ee694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.859808 29903 raft_consensus.cc:2726] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.864315 29949 raft_consensus.cc:482] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.864603 29949 raft_consensus.cc:504] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.865481 29949 leader_election.cc:288] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.865886 28461 tablet_service.cc:1467] Processing CreateTablet for tablet abb9279e91124fe6bfbdc8856eccfed8 (DEFAULT_TABLE table=default.Table_c1936a618b9f488fbd38c45bbc95303c [id=1654ff04429e4f1397b299342743bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.866416 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb9279e91124fe6bfbdc8856eccfed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.872843 28456 tablet_service.cc:1467] Processing CreateTablet for tablet bf9e98bee96d4df895eaed639d033b21 (DEFAULT_TABLE table=default.Table_7e8c0ee351544a32a03b6aa78c2f9264 [id=20cebd5f519640beac7b6ac35e3be8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.873317 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9e98bee96d4df895eaed639d033b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.874413 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3ae54bb8b34f4983e32a5ae664dd66 (DEFAULT_TABLE table=default.Table_c6d80a93009d467ab3bc0bc3c3ba529f [id=13124c075b8841d6a27878706e9c9ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.874783 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3ae54bb8b34f4983e32a5ae664dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.876361 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac145bcc44b4437b840df389cd254f9 (DEFAULT_TABLE table=default.Table_e5c58148b8e649f482d8025f0d340f3b [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.876721 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac145bcc44b4437b840df389cd254f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.881475 28469 tablet_service.cc:1467] Processing CreateTablet for tablet b9263bbe2d804acc967cdf4ff15edd35 (DEFAULT_TABLE table=default.Table_fc8c9ef982c344d697bc3293363f14d1 [id=b2c1350ac4ff47c4b7723a805b69d970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.882030 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9263bbe2d804acc967cdf4ff15edd35. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:22.887764 28427 leader_election.cc:341] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.888042 28466 tablet_service.cc:1467] Processing CreateTablet for tablet ed8cda887ec7497c9950c182e5c2852b (DEFAULT_TABLE table=default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.893630 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8cda887ec7497c9950c182e5c2852b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:22.894199 28427 leader_election.cc:341] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:22.894367 28427 leader_election.cc:341] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 9011de10a7ff497fb35ccb3ada7315ef
I20240503 14:12:22.894519 28427 leader_election.cc:302] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:22.894735 29750 raft_consensus.cc:2781] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.894922 29750 raft_consensus.cc:686] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:22.895148 29750 consensus_queue.cc:237] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.888751 28604 tablet_bootstrap.cc:492] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:22.889704 28473 tablet_service.cc:1467] Processing CreateTablet for tablet ffe63f0b6e1341089f465f3e37da419c (DEFAULT_TABLE table=default.Table_0d6cc797740e4f18a7aad22805859cd5 [id=1a3c8687e5d84377a3215cb23362f78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.897433 28604 ts_tablet_manager.cc:1367] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20240503 14:12:22.897781 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe63f0b6e1341089f465f3e37da419c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.890022 28464 tablet_service.cc:1467] Processing CreateTablet for tablet a7cbb1ecc2294e27a41b61693ac21e9a (DEFAULT_TABLE table=default.Table_f526ebddbfd04608bdf15d7af168e12a [id=81571ea109384d17b612344b854e47dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.900533 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cbb1ecc2294e27a41b61693ac21e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.900931 28604 raft_consensus.cc:348] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.902186 28653 ts_tablet_manager.cc:1888] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:22.891564 28456 tablet_service.cc:1467] Processing CreateTablet for tablet ea5221fc8d6548e08e4ae2c2d466f970 (DEFAULT_TABLE table=default.Table_04ae17bd98c04c4b993d808882cdc3ec [id=5629316959054b54a63bac11c6d3d8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.904214 28604 raft_consensus.cc:374] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.904641 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5221fc8d6548e08e4ae2c2d466f970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.904670 28604 raft_consensus.cc:729] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.905365 29903 raft_consensus.cc:482] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.907181 29903 raft_consensus.cc:504] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.907894 29903 leader_election.cc:288] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.908644 28653 log.cc:1196] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/dda39f0323ec47799d3d44c96f3ba605
I20240503 14:12:22.909662 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6b50aed01a4bf8af8889036dd83606" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:22.910068 28653 ts_tablet_manager.cc:1909] T dda39f0323ec47799d3d44c96f3ba605 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:22.910190 28355 raft_consensus.cc:2454] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:22.910709 28150 leader_election.cc:302] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:22.910944 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 863) took 1036 ms. Trace:
I20240503 14:12:22.905020 28604 consensus_queue.cc:260] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.911077 29903 raft_consensus.cc:2781] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.908692 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6b50aed01a4bf8af8889036dd83606" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:22.911147 29903 raft_consensus.cc:482] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.911201 29903 raft_consensus.cc:3037] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.911350 28493 raft_consensus.cc:2454] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:22.912918 28604 ts_tablet_manager.cc:1398] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:12:22.913213 28604 tablet_bootstrap.cc:492] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:22.913856 29903 raft_consensus.cc:504] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.914582 29903 leader_election.cc:288] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:22.915324 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6b50aed01a4bf8af8889036dd83606" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:22.915324 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6b50aed01a4bf8af8889036dd83606" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:22.915516 28493 raft_consensus.cc:3037] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.915516 28355 raft_consensus.cc:3037] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.918200 28493 raft_consensus.cc:2454] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:22.918196 28355 raft_consensus.cc:2454] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:22.919027 28150 leader_election.cc:302] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:22.919356 29903 raft_consensus.cc:2781] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.919627 29903 raft_consensus.cc:686] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:22.919955 29903 consensus_queue.cc:237] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:22.920487 28427 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0af0342e0e0b47499a9322b5b2cb3d03
W20240503 14:12:22.921058 28427 consensus_peers.cc:485] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95c97a43bd714da68c1a77408f3d1cd1. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:22.923511 28427 consensus_peers.cc:485] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:22.911023 28653 rpcz_store.cc:276] 0503 14:12:21.874446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.876962 (+ 2516us) service_pool.cc:225] Handling call
0503 14:12:22.708887 (+831925us) ts_tablet_manager.cc:1120] Deleting tablet dda39f0323ec47799d3d44c96f3ba605
0503 14:12:22.708894 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.738878 (+ 29984us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.902070 (+163192us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:22.910924 (+ 8854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":822140}
I20240503 14:12:22.926376 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]) successfully deleted
I20240503 14:12:22.926643 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 852) took 1071 ms. Trace:
I20240503 14:12:22.928227 28614 tablet_bootstrap.cc:492] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.928381 28614 ts_tablet_manager.cc:1367] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.103s user 0.037s sys 0.032s
I20240503 14:12:22.929051 28614 raft_consensus.cc:348] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.930145 28614 raft_consensus.cc:374] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.930222 28614 raft_consensus.cc:729] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.930047 28653 rpcz_store.cc:276] 0503 14:12:21.855312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.882948 (+ 27636us) service_pool.cc:225] Handling call
0503 14:12:22.926396 (+1043448us) ts_tablet_manager.cc:1120] Deleting tablet 6874d970e4354070b11060c6b861d7bf
0503 14:12:22.926406 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.926621 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035930}
I20240503 14:12:22.930418 28614 consensus_queue.cc:260] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.930732 28604 tablet_bootstrap.cc:654] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.930919 28614 ts_tablet_manager.cc:1398] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:22.931206 28614 tablet_bootstrap.cc:492] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:22.931366 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6874d970e4354070b11060c6b861d7bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6874d970e4354070b11060c6b861d7bf already in progress: creating tablet
I20240503 14:12:22.931803 28653 tablet_replica.cc:330] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:22.932915 28614 tablet_bootstrap.cc:654] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.933303 28600 tablet_bootstrap.cc:492] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
W20240503 14:12:22.933454 28427 consensus_peers.cc:485] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:22.933517 28600 ts_tablet_manager.cc:1367] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.118s user 0.024s sys 0.057s
I20240503 14:12:22.932919 28653 raft_consensus.cc:2229] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.933864 28653 raft_consensus.cc:2258] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.934396 28600 raft_consensus.cc:348] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.934697 28600 raft_consensus.cc:374] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.934804 28600 raft_consensus.cc:729] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.935204 28427 leader_election.cc:302] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:22.935124 28600 consensus_queue.cc:260] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.935583 29955 raft_consensus.cc:2781] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.935659 29955 raft_consensus.cc:482] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.935714 29955 raft_consensus.cc:3037] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.935724 28600 ts_tablet_manager.cc:1398] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:22.936036 28600 tablet_bootstrap.cc:492] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.937714 28600 tablet_bootstrap.cc:654] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.938154 29955 raft_consensus.cc:504] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.938338 28614 tablet_bootstrap.cc:492] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.938475 28614 ts_tablet_manager.cc:1367] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240503 14:12:22.938843 29955 leader_election.cc:288] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:22.939173 28614 raft_consensus.cc:348] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.939355 28614 raft_consensus.cc:374] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.939415 28614 raft_consensus.cc:729] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.939617 28614 consensus_queue.cc:260] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.707655 29828 raft_consensus.cc:2229] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:22.940147 28614 ts_tablet_manager.cc:1398] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 14:12:22.940157 28427 leader_election.cc:341] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 9011de10a7ff497fb35ccb3ada7315ef
I20240503 14:12:22.940362 28653 ts_tablet_manager.cc:1875] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.940313 29828 raft_consensus.cc:2258] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:22.940433 28614 tablet_bootstrap.cc:492] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.940981 29828 ts_tablet_manager.cc:1875] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:22.942318 28614 tablet_bootstrap.cc:654] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.944185 28427 leader_election.cc:302] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.944648 29955 raft_consensus.cc:2726] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 14:12:22.945359 28427 leader_election.cc:302] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.945747 29955 raft_consensus.cc:2726] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:22.947840 28427 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:22.948114 28614 tablet_bootstrap.cc:492] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:22.948118 28600 tablet_bootstrap.cc:492] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:22.948036 28427 leader_election.cc:302] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:22.948276 28600 ts_tablet_manager.cc:1367] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:12:22.948275 28614 ts_tablet_manager.cc:1367] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 14:12:22.948470 29955 raft_consensus.cc:2726] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.949039 28600 raft_consensus.cc:348] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.949280 28600 raft_consensus.cc:374] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.949338 28600 raft_consensus.cc:729] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.949698 28604 tablet_bootstrap.cc:492] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:22.949568 28600 consensus_queue.cc:260] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.949820 28604 ts_tablet_manager.cc:1367] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 14:12:22.950137 28600 ts_tablet_manager.cc:1398] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:22.950430 28600 tablet_bootstrap.cc:492] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:22.950520 28604 raft_consensus.cc:348] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.950693 28604 raft_consensus.cc:374] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.950744 28604 raft_consensus.cc:729] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.950908 28604 consensus_queue.cc:260] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.951393 28614 raft_consensus.cc:348] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.951581 28614 raft_consensus.cc:374] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.951633 28614 raft_consensus.cc:729] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.951839 28614 consensus_queue.cc:260] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:22.952399 28614 ts_tablet_manager.cc:1398] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:22.952912 28600 tablet_bootstrap.cc:654] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.953128 28197 tablet_service.cc:1467] Processing CreateTablet for tablet b9263bbe2d804acc967cdf4ff15edd35 (DEFAULT_TABLE table=default.Table_fc8c9ef982c344d697bc3293363f14d1 [id=b2c1350ac4ff47c4b7723a805b69d970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.953240 28614 tablet_bootstrap.cc:492] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:22.953755 28604 ts_tablet_manager.cc:1398] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:12:22.954226 28604 tablet_bootstrap.cc:492] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:22.955680 28604 tablet_bootstrap.cc:654] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.957199 28614 tablet_bootstrap.cc:654] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:22.961028 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 780b53efefcf42569bba2fcabd21e211 consensus state has no leader
W20240503 14:12:22.964922 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 244718a7d40d4ad0ac7c3a41bdb325d3 consensus state has no leader
I20240503 14:12:22.965633 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 00a56d79c36f4ffa9bae02e2369d8ded
I20240503 14:12:22.966271 28025 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1037) took 1122 ms. Trace:
I20240503 14:12:22.966665 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a56d79c36f4ffa9bae02e2369d8ded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:22.966388 28025 rpcz_store.cc:276] 0503 14:12:21.843951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.849507 (+ 5556us) service_pool.cc:225] Handling call
0503 14:12:21.850013 (+ 506us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:21.850147 (+ 134us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:21.850204 (+ 57us) catalog_manager.cc:2758] Locking tablets
0503 14:12:22.733049 (+882845us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:22.965423 (+232374us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:22.965520 (+ 97us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:22.965568 (+ 48us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:22.966227 (+ 659us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:22.813858 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:22.813990 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:22.813997 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:22.813999 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:22.814168 (+ 169us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:22.814181 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:22.814186 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:22.814313 (+ 127us) tablet.cc:699] Row locks acquired
0503 14:12:22.814315 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:22.814360 (+ 45us) write_op.cc:270] Start()
0503 14:12:22.814397 (+ 37us) write_op.cc:276] Timestamp: P: 1714745542814315 usec, L: 0
0503 14:12:22.814401 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:22.814609 (+ 208us) log.cc:842] Serialized 731 byte log entry
0503 14:12:22.962743 (+148134us) op_driver.cc:444] REPLICATION: finished
0503 14:12:22.963663 (+ 920us) write_op.cc:301] APPLY: starting
0503 14:12:22.963674 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:22.963723 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:22.963748 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:22.963808 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:22.963837 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:22.963845 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:22.963897 (+ 52us) log.cc:842] Serialized 29 byte log entry
0503 14:12:22.963936 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:22.963951 (+ 15us) write_op.cc:454] Released schema lock
0503 14:12:22.963957 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:22.964008 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:22.964012 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":883,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":390,"num_ops":2,"prepare.queue_time_us":79787,"prepare.run_cpu_time_us":866,"prepare.run_wall_time_us":2270,"replication_time_us":148319,"threads_started":1}]]}
I20240503 14:12:22.972362 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4dc732bcb44222b4b15e5d77defe33" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:22.972631 28212 raft_consensus.cc:3037] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.975663 28212 raft_consensus.cc:2454] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:22.980902 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 95a74f927c974e8aa72ba1462f78116c (DEFAULT_TABLE table=default.Table_664e113b7fe34816ad9b995b341c40ef [id=376eb95d389d4731aca8acc0b1617a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.981297 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9263bbe2d804acc967cdf4ff15edd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.991251 29915 tablet.cc:1722] T 02dd4c914a4846bab3a00408247ec64a P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:22.997537 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 380f267a984541c7878eeaa4eeee8f34
I20240503 14:12:22.998203 28030 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1035) took 1155 ms. Trace:
I20240503 14:12:22.998481 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f267a984541c7878eeaa4eeee8f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:22.998384 28030 rpcz_store.cc:276] 0503 14:12:21.842780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.845305 (+ 2525us) service_pool.cc:225] Handling call
0503 14:12:21.845847 (+ 542us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:21.846003 (+ 156us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:21.846063 (+ 60us) catalog_manager.cc:2758] Locking tablets
0503 14:12:22.734283 (+888220us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:22.997275 (+262992us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:22.997365 (+ 90us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:22.997414 (+ 49us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:22.998135 (+ 721us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:22.816167 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:22.816262 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:22.816267 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:22.816269 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:22.816363 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:22.816372 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:22.816376 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:22.816456 (+ 80us) tablet.cc:699] Row locks acquired
0503 14:12:22.816458 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:22.816491 (+ 33us) write_op.cc:270] Start()
0503 14:12:22.816516 (+ 25us) write_op.cc:276] Timestamp: P: 1714745542816449 usec, L: 0
0503 14:12:22.816520 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:22.816665 (+ 145us) log.cc:842] Serialized 731 byte log entry
0503 14:12:22.995991 (+179326us) op_driver.cc:444] REPLICATION: finished
0503 14:12:22.996156 (+ 165us) write_op.cc:301] APPLY: starting
0503 14:12:22.996176 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:22.996229 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:22.996279 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:22.996363 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:22.996405 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:22.996412 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:22.996498 (+ 86us) log.cc:842] Serialized 29 byte log entry
0503 14:12:22.996568 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:22.996591 (+ 23us) write_op.cc:454] Released schema lock
0503 14:12:22.996599 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:22.996661 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:22.996665 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":111,"apply.run_cpu_time_us":579,"apply.run_wall_time_us":579,"num_ops":2,"prepare.queue_time_us":81132,"prepare.run_cpu_time_us":567,"prepare.run_wall_time_us":890,"replication_time_us":179444}]]}
I20240503 14:12:22.998057 28427 leader_election.cc:302] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:22.999636 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0c300b1f104949b3ae74f5b38d378a11
I20240503 14:12:22.999969 29955 raft_consensus.cc:2781] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.000233 29955 raft_consensus.cc:686] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:23.000491 29955 consensus_queue.cc:237] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.001480 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c300b1f104949b3ae74f5b38d378a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:23.006203 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_06fe0125c0644021bf434b53126d7f3a" } modify_external_catalogs: true
I20240503 14:12:23.006492 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_06fe0125c0644021bf434b53126d7f3a" } modify_external_catalogs: true
I20240503 14:12:23.007928 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_acbb6ef1f5a84c11847e5299afebb8b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.008581 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acbb6ef1f5a84c11847e5299afebb8b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.009768 29921 tablet.cc:1722] T 02dd4c914a4846bab3a00408247ec64a P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:23.009959 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9af4e42a49be4a0a877834654f2171af" } new_table_name: "default.Table_7cdef12fca924e8da0854d0042ec0e05" modify_external_catalogs: true
I20240503 14:12:23.011322 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0b5dae96b3b64066ba7334dc73e578e8" } new_table_name: "default.Table_33e73e5cf1674fcf969ae2750f89d336" modify_external_catalogs: true
I20240503 14:12:23.012874 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e0e6e0f2ed4d4e67aef27d3d4337db9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.013442 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e6e0f2ed4d4e67aef27d3d4337db9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.023561 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 86804987bdf340dc83ddd58ba2bc8a96
I20240503 14:12:23.024407 28031 catalog_manager.cc:6554] ReplaceTablet: tablet 86804987bdf340dc83ddd58ba2bc8a96 deleted and replaced by tablet addc2a6195034b0a927fdee7a2955122
I20240503 14:12:23.024572 28031 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1033) took 1257 ms. Trace:
I20240503 14:12:23.024554 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 86804987bdf340dc83ddd58ba2bc8a96 with delete_type TABLET_DATA_DELETED (replaced by tablet addc2a6195034b0a927fdee7a2955122) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:23.024710 28031 rpcz_store.cc:276] 0503 14:12:21.767099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.838484 (+ 71385us) service_pool.cc:225] Handling call
0503 14:12:23.024510 (+1186026us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.015604 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.015755 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.015762 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:23.015765 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.015878 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.015891 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.015896 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.016004 (+ 108us) tablet.cc:699] Row locks acquired
0503 14:12:23.016006 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:23.016048 (+ 42us) write_op.cc:270] Start()
0503 14:12:23.016399 (+ 351us) write_op.cc:276] Timestamp: P: 1714745543016031 usec, L: 0
0503 14:12:23.016405 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.017458 (+ 1053us) log.cc:842] Serialized 628 byte log entry
0503 14:12:23.022563 (+ 5105us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.022640 (+ 77us) write_op.cc:301] APPLY: starting
0503 14:12:23.022655 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.022703 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.022748 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.022893 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.022964 (+ 71us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:23.022971 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:23.023045 (+ 74us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.023122 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.023143 (+ 21us) write_op.cc:454] Released schema lock
0503 14:12:23.023150 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.023207 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.023211 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":619,"num_ops":2,"prepare.queue_time_us":276068,"replication_time_us":6132}]]}
I20240503 14:12:23.027041 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8220e8ba6ccb4f89ba7481c40f8af3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.027724 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8220e8ba6ccb4f89ba7481c40f8af3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.029615 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c300b1f104949b3ae74f5b38d378a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:23.030066 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 86804987bdf340dc83ddd58ba2bc8a96 with delete_type TABLET_DATA_DELETED (replaced by tablet addc2a6195034b0a927fdee7a2955122) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:23.035677 28653 ts_tablet_manager.cc:1888] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:23.036393 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a56d79c36f4ffa9bae02e2369d8ded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:23.039080 28653 log.cc:1196] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/41cbd1326869434594e126421ad132d8
I20240503 14:12:23.039645 28653 ts_tablet_manager.cc:1909] T 41cbd1326869434594e126421ad132d8 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:23.043751 28028 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 02dd4c914a4846bab3a00408247ec64a (table default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc]). Expected version 2 got 0
I20240503 14:12:23.045082 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 864) took 1098 ms. Trace:
I20240503 14:12:23.045184 28653 rpcz_store.cc:276] 0503 14:12:21.946265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.946321 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:22.931750 (+985429us) ts_tablet_manager.cc:1120] Deleting tablet 41cbd1326869434594e126421ad132d8
0503 14:12:22.931761 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.943756 (+ 11995us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.035542 (+ 91786us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.045052 (+ 9510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":985241}
I20240503 14:12:23.045675 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 41cbd1326869434594e126421ad132d8 (table default.Table_a07e2f0650084270860bd28e40f224aa [id=a98b2a6e32bf425eba5d2e039def6b0f]) successfully deleted
I20240503 14:12:23.045773 28029 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_b4868a6959c44bfbaad9b187dd171f41 [id=0114799f37994a3396ee46fd1f2e98b5] alter complete (version 2)
I20240503 14:12:23.045912 28653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 865) took 1099 ms. Trace:
I20240503 14:12:23.045994 28653 rpcz_store.cc:276] 0503 14:12:21.946721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.946767 (+ 46us) service_pool.cc:225] Handling call
0503 14:12:23.045698 (+1098931us) ts_tablet_manager.cc:1120] Deleting tablet b04fff656f624001a709f783816eec1e
0503 14:12:23.045710 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.045887 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1098773}
W20240503 14:12:23.046442 28029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.193:45997 (request call id 94) took 826 ms (client timeout 1000 ms). Trace:
W20240503 14:12:23.046510 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b04fff656f624001a709f783816eec1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b04fff656f624001a709f783816eec1e already in progress: creating tablet
W20240503 14:12:23.046571 28029 rpcz_store.cc:269] 0503 14:12:22.219445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.606990 (+387545us) service_pool.cc:225] Handling call
0503 14:12:22.633896 (+ 26906us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.046305 (+412409us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:22.973308 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:22.973446 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:22.973454 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:22.973457 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:22.973584 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:22.973599 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:22.973605 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:22.973677 (+ 72us) tablet.cc:699] Row locks acquired
0503 14:12:22.973679 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:22.981865 (+ 8186us) write_op.cc:270] Start()
0503 14:12:22.981920 (+ 55us) write_op.cc:276] Timestamp: P: 1714745542981829 usec, L: 0
0503 14:12:22.981926 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.005412 (+ 23486us) log.cc:842] Serialized 356 byte log entry
0503 14:12:23.015155 (+ 9743us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 000000000095519a 00007fe125139162 00007fe1264a1e32 00007fe1264a0959 00007fe1264a886b 00007fe1264a86e8 0000000000a60bcc 00007fe11993ab74 00007fe11993e19b 00007fe11993e018 0000000000a60bcc
0503 14:12:23.018478 (+ 3323us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.018562 (+ 84us) write_op.cc:301] APPLY: starting
0503 14:12:23.018579 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.018621 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.018624 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.018678 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.018723 (+ 45us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:23.018731 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:23.018808 (+ 77us) log.cc:842] Serialized 23 byte log entry
0503 14:12:23.018874 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.018892 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:23.018899 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.018954 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.018958 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":60284288,"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":443,"apply.run_wall_time_us":443,"num_ops":1,"prepare.queue_time_us":234911,"prepare.run_cpu_time_us":1228,"prepare.run_wall_time_us":42273,"raft.queue_time_us":39,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":262,"replication_time_us":36526,"spinlock_wait_cycles":53513600}]]}
I20240503 14:12:23.047717 28600 tablet_bootstrap.cc:492] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:23.047869 28600 ts_tablet_manager.cc:1367] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.098s user 0.013s sys 0.031s
W20240503 14:12:23.048226 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 00a56d79c36f4ffa9bae02e2369d8ded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00a56d79c36f4ffa9bae02e2369d8ded already in progress: creating tablet
W20240503 14:12:23.048640 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 380f267a984541c7878eeaa4eeee8f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 380f267a984541c7878eeaa4eeee8f34 already in progress: creating tablet
I20240503 14:12:23.048923 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ed4dc732bcb44222b4b15e5d77defe33
I20240503 14:12:23.048979 28604 tablet_bootstrap.cc:492] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
W20240503 14:12:23.048977 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0c300b1f104949b3ae74f5b38d378a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c300b1f104949b3ae74f5b38d378a11 already in progress: creating tablet
I20240503 14:12:23.049121 28604 ts_tablet_manager.cc:1367] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.095s user 0.034s sys 0.047s
I20240503 14:12:23.049710 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet ffcfc89a415e4c68b6a54318755fe0d7 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:23.049932 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:23.049877 28604 raft_consensus.cc:348] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:23.050318 28026 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 84) took 979 ms (client timeout 1000 ms). Trace:
I20240503 14:12:23.050383 28600 raft_consensus.cc:348] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:23.050446 28026 rpcz_store.cc:269] 0503 14:12:22.070761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.143396 (+ 72635us) service_pool.cc:225] Handling call
0503 14:12:23.050219 (+906823us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20240503 14:12:23.050570 28600 raft_consensus.cc:374] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.050616 28600 raft_consensus.cc:729] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.050925 28600 consensus_queue.cc:260] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.051587 28600 ts_tablet_manager.cc:1398] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:23.052039 28604 raft_consensus.cc:374] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.052134 28604 raft_consensus.cc:729] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.052368 28604 consensus_queue.cc:260] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:23.053834 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 86804987bdf340dc83ddd58ba2bc8a96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86804987bdf340dc83ddd58ba2bc8a96 already in progress: creating tablet
I20240503 14:12:23.055848 28604 ts_tablet_manager.cc:1398] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:23.056731 28653 tablet_replica.cc:330] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:23.056972 28653 raft_consensus.cc:2229] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:23.057152 28653 raft_consensus.cc:2258] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:23.058064 28604 tablet_bootstrap.cc:492] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:23.058704 28600 tablet_bootstrap.cc:492] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:23.059626 28604 tablet_bootstrap.cc:654] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.060868 28600 tablet_bootstrap.cc:654] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.062371 28653 ts_tablet_manager.cc:1875] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:23.066610 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:23.067329 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f267a984541c7878eeaa4eeee8f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:23.068632 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_905413b2b2024c9685068fe4c869f748"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.069280 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_905413b2b2024c9685068fe4c869f748 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.070914 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d8f7551fc7ad4ea28e5a730ab6f2d0d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.071579 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8f7551fc7ad4ea28e5a730ab6f2d0d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.073064 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a74f927c974e8aa72ba1462f78116c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.074602 28653 ts_tablet_manager.cc:1888] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:23.075353 28327 tablet_service.cc:1467] Processing CreateTablet for tablet a526e9f1e9934e519349dcd4523ee694 (DEFAULT_TABLE table=default.Table_55224d5639254fa1a4890def9934e40a [id=84b9cf0bbc494213a97d9f1aa7ce30ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.075490 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda92c8e3c94139b84eabc81127613e (DEFAULT_TABLE table=default.Table_463c966bd4c847d0a0967bc7dae04ec6 [id=5c1c5cfd2cef4bd6ad984e404026d90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.075800 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a526e9f1e9934e519349dcd4523ee694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.075994 28323 tablet_service.cc:1467] Processing CreateTablet for tablet ceec6992b0474706b8b3766518b0c274 (DEFAULT_TABLE table=default.Table_decc999a1649405b8e7301584192c536 [id=ccb4698582a54cdcac8066d50c7af7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.076366 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceec6992b0474706b8b3766518b0c274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.076850 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda92c8e3c94139b84eabc81127613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.083400 28653 log.cc:1196] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ed4dc732bcb44222b4b15e5d77defe33
I20240503 14:12:23.084520 28653 ts_tablet_manager.cc:1909] T ed4dc732bcb44222b4b15e5d77defe33 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:23.085455 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 5500da0e5658492b840dd29273ab265f (DEFAULT_TABLE table=default.Table_70df8ea2254f45aab971123dd7f55a44 [id=24611f8249494aa094924164a4f2c1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.085995 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5500da0e5658492b840dd29273ab265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.086190 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ed4dc732bcb44222b4b15e5d77defe33 (table default.Table_3ce7004b7c164629a92e6bcbb421348d [id=a6a9ad0f171448c2a3d3401e02ec62c0]) successfully deleted
I20240503 14:12:23.086548 29876 catalog_manager.cc:1259] Loaded cluster ID: 09434ad4cf0745ecb5d037e3cd83df07
I20240503 14:12:23.086750 29876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: loading cluster ID for follower catalog manager: success
I20240503 14:12:23.091719 28334 tablet_service.cc:1467] Processing CreateTablet for tablet ffe63f0b6e1341089f465f3e37da419c (DEFAULT_TABLE table=default.Table_0d6cc797740e4f18a7aad22805859cd5 [id=1a3c8687e5d84377a3215cb23362f78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.092272 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe63f0b6e1341089f465f3e37da419c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.093034 28318 tablet_service.cc:1467] Processing CreateTablet for tablet bf9e98bee96d4df895eaed639d033b21 (DEFAULT_TABLE table=default.Table_7e8c0ee351544a32a03b6aa78c2f9264 [id=20cebd5f519640beac7b6ac35e3be8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.093465 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9e98bee96d4df895eaed639d033b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.098973 28327 tablet_service.cc:1467] Processing CreateTablet for tablet abb9279e91124fe6bfbdc8856eccfed8 (DEFAULT_TABLE table=default.Table_c1936a618b9f488fbd38c45bbc95303c [id=1654ff04429e4f1397b299342743bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.099506 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abb9279e91124fe6bfbdc8856eccfed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.102150 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac145bcc44b4437b840df389cd254f9 (DEFAULT_TABLE table=default.Table_e5c58148b8e649f482d8025f0d340f3b [id=c852c557af404f10a65b725b5d50ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.102581 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ac145bcc44b4437b840df389cd254f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.113204 29903 raft_consensus.cc:482] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.113391 29903 raft_consensus.cc:504] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.114133 29903 leader_election.cc:288] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:23.115054 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94673f00cb3432185009bda2ce398fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:23.115922 28150 leader_election.cc:341] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:23.123090 28323 tablet_service.cc:1467] Processing CreateTablet for tablet b9263bbe2d804acc967cdf4ff15edd35 (DEFAULT_TABLE table=default.Table_fc8c9ef982c344d697bc3293363f14d1 [id=b2c1350ac4ff47c4b7723a805b69d970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.123545 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9263bbe2d804acc967cdf4ff15edd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.124662 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94673f00cb3432185009bda2ce398fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:23.124954 28355 raft_consensus.cc:2454] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:23.125523 28150 leader_election.cc:302] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.126120 29903 raft_consensus.cc:2781] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.126212 29903 raft_consensus.cc:482] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.126276 29903 raft_consensus.cc:3037] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.126698 29876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: acquiring CA information for follower catalog manager: success
I20240503 14:12:23.127279 28331 tablet_service.cc:1467] Processing CreateTablet for tablet a7cbb1ecc2294e27a41b61693ac21e9a (DEFAULT_TABLE table=default.Table_f526ebddbfd04608bdf15d7af168e12a [id=81571ea109384d17b612344b854e47dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.127316 28336 tablet_service.cc:1467] Processing CreateTablet for tablet ed8cda887ec7497c9950c182e5c2852b (DEFAULT_TABLE table=default.Table_151ead7d1ad1474b8facf3f42c1cdac1 [id=00e0cb75e090416ea7903b7176c57f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.127715 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cbb1ecc2294e27a41b61693ac21e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.127749 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a56d79c36f4ffa9bae02e2369d8ded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:23.128209 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8cda887ec7497c9950c182e5c2852b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.130931 29903 raft_consensus.cc:504] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.132534 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94673f00cb3432185009bda2ce398fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:23.133102 28150 leader_election.cc:341] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:23.133554 29903 leader_election.cc:288] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:23.135883 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f267a984541c7878eeaa4eeee8f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:23.135958 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c300b1f104949b3ae74f5b38d378a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:23.136449 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 86804987bdf340dc83ddd58ba2bc8a96 with delete_type TABLET_DATA_DELETED (replaced by tablet addc2a6195034b0a927fdee7a2955122) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:23.140525 28323 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:23.141899 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084bd4fbf604042bb33ff03c3f8cc19" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.142192 28493 raft_consensus.cc:2454] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:23.143409 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084bd4fbf604042bb33ff03c3f8cc19" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.143638 28213 raft_consensus.cc:2454] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:23.149883 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94673f00cb3432185009bda2ce398fa" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:23.150279 28355 raft_consensus.cc:3037] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.151124 29864 raft_consensus.cc:482] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.151329 29864 raft_consensus.cc:504] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.152122 29864 leader_election.cc:288] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.172820 29876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:12:23.177906 29980 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:12:23.178203 29980 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:12:23.179342 29980 leader_election.cc:288] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:12:23.180150 29486 ts_manager.cc:194] Registered new tserver with Master: d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:23.182943 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1109 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:12:23.183337 28898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1109 } } ignore_live_leader: false dest_uuid: "bb6a7aee99214df4b02858ddae025b14" is_pre_election: true
I20240503 14:12:23.186057 29436 leader_election.cc:302] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc; no voters: bb6a7aee99214df4b02858ddae025b14, c0690a516c89446d85c88cdf64fa4238
I20240503 14:12:23.193174 29980 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:23.221802 28355 raft_consensus.cc:2454] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:23.225250 28150 leader_election.cc:302] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.225950 29903 raft_consensus.cc:2781] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.226543 29903 raft_consensus.cc:686] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:23.226964 29903 consensus_queue.cc:237] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.243407 28317 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58036 (request call id 134) took 1183 ms. Trace:
I20240503 14:12:23.266599 28317 rpcz_store.cc:276] 0503 14:12:22.060033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.060172 (+ 139us) service_pool.cc:225] Handling call
0503 14:12:22.332244 (+272072us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x616000211054. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52ca1aeb00 00007f52cdec0b7f 00007f52c884c8e6 00007f52c884c615 00007f52c567804e 00007f52c5676f2e 00007f52c567d405 00007f52c567dd2f 00007f52c567db58 0000000000a60bcc
0503 14:12:22.410971 (+ 78727us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x6160000a3f54. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52ca1aeb00 00007f52cdec0b7f 00007f52c884c8e6 00007f52c884c615 00007f52c567804e 00007f52c5676f2e 00007f52c567d405 00007f52c567dd2f 00007f52c567db58 0000000000a60bcc
0503 14:12:22.578706 (+167735us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x616000221b54. stack: 00007f52be970756 00007f52be970510 00007f52bd9a348d 0000000000955804 00000000009557cd 000000000095519a 00007f52ca1aeb00 00007f52cdec0b7f 00007f52c884c8e6 00007f52c884c615 00007f52c567804e 00007f52c5676f2e 00007f52c567d405 00007f52c567dd2f 00007f52c567db58 0000000000a60bcc
0503 14:12:23.243361 (+664655us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":781442048}
I20240503 14:12:23.123131 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3ae54bb8b34f4983e32a5ae664dd66 (DEFAULT_TABLE table=default.Table_c6d80a93009d467ab3bc0bc3c3ba529f [id=13124c075b8841d6a27878706e9c9ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.264753 29864 raft_consensus.cc:482] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.269173 29864 raft_consensus.cc:504] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.270031 29864 leader_election.cc:288] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.265306 29949 raft_consensus.cc:482] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.270298 29949 raft_consensus.cc:504] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.270874 29949 leader_election.cc:288] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.265986 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4dc732bcb44222b4b15e5d77defe33" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:23.271086 28353 raft_consensus.cc:3037] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.271714 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3ae54bb8b34f4983e32a5ae664dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.277294 29828 ts_tablet_manager.cc:1888] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:23.277565 29828 log.cc:1196] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/5dd8b2adaeeb4c3abfbfe68cc005e03a
I20240503 14:12:23.277966 29828 ts_tablet_manager.cc:1909] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:23.279480 28661 ts_tablet_manager.cc:1888] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:23.279688 28661 log.cc:1196] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/140de4d395d54b7696062ed96b8aa89c
I20240503 14:12:23.280345 28661 ts_tablet_manager.cc:1909] T 140de4d395d54b7696062ed96b8aa89c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:23.281316 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 739) took 4697 ms. Trace:
I20240503 14:12:23.281482 28661 rpcz_store.cc:276] 0503 14:12:18.584071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.584158 (+ 87us) service_pool.cc:225] Handling call
0503 14:12:22.794719 (+4210561us) ts_tablet_manager.cc:1120] Deleting tablet 140de4d395d54b7696062ed96b8aa89c
0503 14:12:22.794729 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.029980 (+235251us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.279331 (+249351us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.281285 (+ 1954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4210363}
I20240503 14:12:23.282514 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 740) took 4670 ms. Trace:
I20240503 14:12:23.284334 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 140de4d395d54b7696062ed96b8aa89c (table default.Table_dbb3e0a033c84560a6458571881b25e5 [id=238df3dccaa54f4f956476ccb58b0816]) successfully deleted
I20240503 14:12:23.291118 28350 raft_consensus.cc:3037] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:23.297102 28019 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 725b4c9a47c0458ba188c0cf62f3fe48 (table default.Table_35e27f7a242d4ffaa19a6e07ec525559 [id=ec20cb1644174d6c8a21107acbf0a040], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 725b4c9a47c0458ba188c0cf62f3fe48 consensus state has no leader
I20240503 14:12:23.297406 29949 raft_consensus.cc:482] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.297581 29949 raft_consensus.cc:504] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.298285 29949 leader_election.cc:288] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.315151 28327 tablet_service.cc:1467] Processing CreateTablet for tablet ea5221fc8d6548e08e4ae2c2d466f970 (DEFAULT_TABLE table=default.Table_04ae17bd98c04c4b993d808882cdc3ec [id=5629316959054b54a63bac11c6d3d8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.319804 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c61b9232ebd4431a3d4202ad971d540" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.320026 28213 raft_consensus.cc:2454] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:23.321972 29949 raft_consensus.cc:482] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.322124 29949 raft_consensus.cc:504] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.322739 29949 leader_election.cc:288] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.324416 28290 leader_election.cc:302] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:23.324788 29949 raft_consensus.cc:2781] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.324869 29949 raft_consensus.cc:482] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.324929 29949 raft_consensus.cc:3037] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.328166 29949 raft_consensus.cc:504] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.329897 28600 tablet_bootstrap.cc:492] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:23.330068 28600 ts_tablet_manager.cc:1367] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.271s user 0.080s sys 0.113s
I20240503 14:12:23.330904 28033 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 7ad0e6824f424b8da36fa9eb648b5acc
I20240503 14:12:23.330869 28600 raft_consensus.cc:348] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.331095 28600 raft_consensus.cc:374] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.331108 28033 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1059) took 1097 ms. Trace:
I20240503 14:12:23.331146 28600 raft_consensus.cc:729] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.331450 28600 consensus_queue.cc:260] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.331218 28033 rpcz_store.cc:276] 0503 14:12:22.233971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.965215 (+731244us) service_pool.cc:225] Handling call
0503 14:12:23.006117 (+ 40902us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedfca6 00007fe123ed1f66 00007fe123ed1c95 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.006594 (+ 477us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:23.006722 (+ 128us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:23.006798 (+ 76us) catalog_manager.cc:2758] Locking tablets
0503 14:12:23.006899 (+ 101us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:23.330652 (+323753us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:23.330782 (+ 130us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:23.330834 (+ 52us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:23.331019 (+ 185us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.067754 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.067919 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.067925 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:23.067927 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:23.109392 (+ 41465us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.109412 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.109420 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.154246 (+ 44826us) tablet.cc:699] Row locks acquired
0503 14:12:23.154250 (+ 4us) write_op.cc:260] PREPARE: finished
0503 14:12:23.154319 (+ 69us) write_op.cc:270] Start()
0503 14:12:23.154366 (+ 47us) write_op.cc:276] Timestamp: P: 1714745543154281 usec, L: 0
0503 14:12:23.154370 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.154719 (+ 349us) log.cc:842] Serialized 533 byte log entry
0503 14:12:23.329385 (+174666us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.329555 (+ 170us) write_op.cc:301] APPLY: starting
0503 14:12:23.329628 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.329682 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.329728 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.329807 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.329851 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:23.329859 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:23.329957 (+ 98us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.330515 (+ 558us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.330545 (+ 30us) write_op.cc:454] Released schema lock
0503 14:12:23.330554 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":93435648,"child_traces":[["op",{"apply.queue_time_us":112,"num_ops":2,"prepare.queue_time_us":60155,"replication_time_us":174977}]]}
I20240503 14:12:23.336477 28600 ts_tablet_manager.cc:1398] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:23.338462 28600 tablet_bootstrap.cc:492] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:23.340942 28600 tablet_bootstrap.cc:654] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.346936 29864 raft_consensus.cc:482] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.347133 29864 raft_consensus.cc:504] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.347852 29864 leader_election.cc:288] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.376556 28600 tablet_bootstrap.cc:492] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:23.376713 28600 ts_tablet_manager.cc:1367] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.021s
I20240503 14:12:23.377650 28600 raft_consensus.cc:348] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.377756 28032 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1058) took 1153 ms. Trace:
I20240503 14:12:23.377853 28600 raft_consensus.cc:374] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.377897 28600 raft_consensus.cc:729] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.378157 28600 consensus_queue.cc:260] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.377900 28032 rpcz_store.cc:276] 0503 14:12:22.224267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.961321 (+737054us) service_pool.cc:225] Handling call
0503 14:12:23.007771 (+ 46450us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.008679 (+ 908us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:23.009217 (+ 538us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:23.377401 (+368184us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:23.377536 (+ 135us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:23.377645 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.358189 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.358331 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.358337 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:23.358340 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.358448 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.358463 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.358468 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.358556 (+ 88us) tablet.cc:699] Row locks acquired
0503 14:12:23.358558 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:23.358603 (+ 45us) write_op.cc:270] Start()
0503 14:12:23.358641 (+ 38us) write_op.cc:276] Timestamp: P: 1714745543358557 usec, L: 0
0503 14:12:23.358645 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.368171 (+ 9526us) log.cc:842] Serialized 433 byte log entry
0503 14:12:23.374494 (+ 6323us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.375006 (+ 512us) write_op.cc:301] APPLY: starting
0503 14:12:23.375025 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.375073 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.375121 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.375236 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.375273 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:23.375280 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:23.375354 (+ 74us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.375602 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.375777 (+ 175us) write_op.cc:454] Released schema lock
0503 14:12:23.375785 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":106106112,"child_traces":[["op",{"apply.queue_time_us":443,"num_ops":2,"prepare.queue_time_us":348499,"prepare.run_cpu_time_us":808,"prepare.run_wall_time_us":11231,"replication_time_us":15825}]]}
I20240503 14:12:23.378895 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_eeead1b7b6d54b15845ec251e33f6ca8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.379591 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eeead1b7b6d54b15845ec251e33f6ca8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.386526 28600 ts_tablet_manager.cc:1398] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:12:23.386866 28600 tablet_bootstrap.cc:492] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:23.387317 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bc05dfe2b324ca787ca33f52be6a308 consensus state has no leader
I20240503 14:12:23.387566 28027 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1062) took 1037 ms. Trace:
I20240503 14:12:23.387676 28027 rpcz_store.cc:276] 0503 14:12:22.349847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.000336 (+650489us) service_pool.cc:225] Handling call
0503 14:12:23.009847 (+ 9511us) spinlock_profiling.cc:243] Waited 9.23 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cee0f46 00007fe123ed3c26 00007fe123ed3955 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.010174 (+ 327us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:23.010370 (+ 196us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:23.010539 (+ 169us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:23.010598 (+ 59us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:23.387031 (+376433us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:23.387479 (+ 448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.369459 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.369583 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.369590 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:23.369593 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.369701 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.369712 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.369717 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:23.369777 (+ 60us) tablet.cc:699] Row locks acquired
0503 14:12:23.369779 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:23.369817 (+ 38us) write_op.cc:270] Start()
0503 14:12:23.369852 (+ 35us) write_op.cc:276] Timestamp: P: 1714745543369773 usec, L: 0
0503 14:12:23.369857 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.370069 (+ 212us) log.cc:842] Serialized 405 byte log entry
0503 14:12:23.385809 (+ 15740us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.385942 (+ 133us) write_op.cc:301] APPLY: starting
0503 14:12:23.385957 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.386004 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.386031 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.386092 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.386151 (+ 59us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:23.386160 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:23.386252 (+ 92us) log.cc:842] Serialized 23 byte log entry
0503 14:12:23.386348 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.386367 (+ 19us) write_op.cc:454] Released schema lock
0503 14:12:23.386374 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.386432 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.386436 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":21238016,"child_traces":[["op",{"apply.queue_time_us":85,"apply.run_cpu_time_us":544,"apply.run_wall_time_us":544,"num_ops":1,"prepare.queue_time_us":358376,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":1056,"replication_time_us":15928}]]}
I20240503 14:12:23.388638 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 920b41e3b21347b393497de4d4e87469 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:23.389150 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2bc7bb4b52984487aa15a8a7e98ae3ca consensus state has no leader
I20240503 14:12:23.389328 28025 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1060) took 1154 ms. Trace:
I20240503 14:12:23.389497 28025 rpcz_store.cc:276] 0503 14:12:22.234936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.967211 (+732275us) service_pool.cc:225] Handling call
0503 14:12:23.011250 (+ 44039us) spinlock_profiling.cc:243] Waited 43.8 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cee0f46 00007fe123ed3c26 00007fe123ed3955 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.011485 (+ 235us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:23.011657 (+ 172us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:23.011856 (+ 199us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:23.011943 (+ 87us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:23.388876 (+376933us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:23.389276 (+ 400us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.370548 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.370640 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.370647 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:23.370649 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:23.370743 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.370753 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.370758 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:23.370812 (+ 54us) tablet.cc:699] Row locks acquired
0503 14:12:23.370814 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:23.370846 (+ 32us) write_op.cc:270] Start()
0503 14:12:23.370874 (+ 28us) write_op.cc:276] Timestamp: P: 1714745543370803 usec, L: 0
0503 14:12:23.370878 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.371062 (+ 184us) log.cc:842] Serialized 405 byte log entry
0503 14:12:23.385876 (+ 14814us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.386522 (+ 646us) write_op.cc:301] APPLY: starting
0503 14:12:23.386534 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.386568 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.386615 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.386657 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.386687 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:23.386704 (+ 17us) write_op.cc:312] APPLY: finished
0503 14:12:23.386762 (+ 58us) log.cc:842] Serialized 23 byte log entry
0503 14:12:23.386806 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.386820 (+ 14us) write_op.cc:454] Released schema lock
0503 14:12:23.386826 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.386873 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.386877 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":100697856,"child_traces":[["op",{"apply.queue_time_us":614,"apply.run_cpu_time_us":397,"apply.run_wall_time_us":396,"num_ops":1,"prepare.queue_time_us":358177,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":870,"replication_time_us":14983}]]}
I20240503 14:12:23.390249 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:23.390941 28600 tablet_bootstrap.cc:654] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.393265 29903 raft_consensus.cc:482] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.393393 29903 raft_consensus.cc:504] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.393996 29903 leader_election.cc:288] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:23.395730 28030 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1061) took 1130 ms. Trace:
I20240503 14:12:23.395901 28030 rpcz_store.cc:276] 0503 14:12:22.265673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.999030 (+733357us) service_pool.cc:225] Handling call
0503 14:12:23.012757 (+ 13727us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.013535 (+ 778us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:23.013996 (+ 461us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:23.395428 (+381432us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:23.395558 (+ 130us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:23.395663 (+ 105us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.371472 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.371558 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.371563 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:23.371565 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:23.371658 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.371667 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.371672 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.371742 (+ 70us) tablet.cc:699] Row locks acquired
0503 14:12:23.371745 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:23.371774 (+ 29us) write_op.cc:270] Start()
0503 14:12:23.371800 (+ 26us) write_op.cc:276] Timestamp: P: 1714745543371732 usec, L: 0
0503 14:12:23.371804 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.371934 (+ 130us) log.cc:842] Serialized 433 byte log entry
0503 14:12:23.392033 (+ 20099us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.394191 (+ 2158us) write_op.cc:301] APPLY: starting
0503 14:12:23.394214 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.394264 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.394312 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.394440 (+ 128us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.394481 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:23.394488 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:23.394583 (+ 95us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.394647 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.394672 (+ 25us) write_op.cc:454] Released schema lock
0503 14:12:23.394679 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.394760 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.394764 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":30930816,"child_traces":[["op",{"apply.queue_time_us":2080,"apply.run_cpu_time_us":640,"apply.run_wall_time_us":640,"num_ops":2,"prepare.queue_time_us":357017,"prepare.run_cpu_time_us":549,"prepare.run_wall_time_us":888,"replication_time_us":20202}]]}
I20240503 14:12:23.396790 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6c996bea4ba44fefbb3bc602c3c76f58" } new_table_name: "default.Table_ebd8ae7785ef42cca8a6b747de4e0285" modify_external_catalogs: true
I20240503 14:12:23.398445 28031 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1063) took 1046 ms. Trace:
I20240503 14:12:23.398597 28031 rpcz_store.cc:276] 0503 14:12:22.351842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.025325 (+673483us) service_pool.cc:225] Handling call
0503 14:12:23.026913 (+ 1588us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.027823 (+ 910us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:23.028378 (+ 555us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:23.398162 (+369784us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:23.398279 (+ 117us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:23.398373 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.372375 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.372460 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.372465 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:23.372467 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:23.372538 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.372547 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.372552 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.372614 (+ 62us) tablet.cc:699] Row locks acquired
0503 14:12:23.372617 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:23.372645 (+ 28us) write_op.cc:270] Start()
0503 14:12:23.372670 (+ 25us) write_op.cc:276] Timestamp: P: 1714745543372603 usec, L: 0
0503 14:12:23.372673 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.372820 (+ 147us) log.cc:842] Serialized 433 byte log entry
0503 14:12:23.392130 (+ 19310us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.394860 (+ 2730us) write_op.cc:301] APPLY: starting
0503 14:12:23.394873 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.394930 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.394955 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.395064 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.395096 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:23.395104 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:23.395168 (+ 64us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.395217 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.395232 (+ 15us) write_op.cc:454] Released schema lock
0503 14:12:23.395238 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.395290 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.395293 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":2974464,"child_traces":[["op",{"apply.queue_time_us":2697,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":476,"num_ops":2,"prepare.queue_time_us":343466,"prepare.run_cpu_time_us":544,"prepare.run_wall_time_us":1214,"replication_time_us":19439}]]}
I20240503 14:12:23.399525 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2746fc24790d430ea83a7a159d11260e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.400179 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2746fc24790d430ea83a7a159d11260e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.405089 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_be835f9c7805488e8c2f84b41440578d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.405722 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be835f9c7805488e8c2f84b41440578d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.408418 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7cd2dac6769a49629f32f98471b10966"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.409024 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cd2dac6769a49629f32f98471b10966 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:23.285708 28427 consensus_peers.cc:485] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 495190fca06a40a386c8215eb2195920. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:23.416478 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 130814us
I20240503 14:12:23.291364 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5297664a4d943c28adfb6ac4f061680" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.417219 28493 raft_consensus.cc:2454] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:23.319190 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c61b9232ebd4431a3d4202ad971d540" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.417809 28488 raft_consensus.cc:2454] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:23.320976 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495190fca06a40a386c8215eb2195920" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.402891 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cd09e769be4bee95bbef9c3c550451" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:23.419903 28150 leader_election.cc:341] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:23.421257 28661 rpcz_store.cc:276] 0503 14:12:18.612369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.612457 (+ 88us) service_pool.cc:225] Handling call
0503 14:12:23.282352 (+4669895us) ts_tablet_manager.cc:1120] Deleting tablet 2d5192083ed94a3d8d2d1a0bf001a1eb
0503 14:12:23.282365 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.282492 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4669640}
I20240503 14:12:23.421898 28661 tablet_replica.cc:330] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:23.421849 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2d5192083ed94a3d8d2d1a0bf001a1eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d5192083ed94a3d8d2d1a0bf001a1eb already in progress: creating tablet
I20240503 14:12:23.422216 28661 raft_consensus.cc:2229] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:23.422475 28661 raft_consensus.cc:2258] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:23.423117 28661 ts_tablet_manager.cc:1875] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:23.427719 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 103326us
W20240503 14:12:23.428202 28290 consensus_peers.cc:485] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41cbd1326869434594e126421ad132d8. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:23.428463 28290 leader_election.cc:341] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: c5297664a4d943c28adfb6ac4f061680
I20240503 14:12:23.429203 28290 leader_election.cc:302] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:23.429606 29864 raft_consensus.cc:2781] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.429687 29864 raft_consensus.cc:482] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.429760 29864 raft_consensus.cc:3037] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.434502 28353 raft_consensus.cc:2454] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:23.394747 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cd09e769be4bee95bbef9c3c550451" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:23.435739 28349 raft_consensus.cc:2454] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:23.436506 28150 leader_election.cc:302] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.436998 29903 raft_consensus.cc:2781] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.437080 29903 raft_consensus.cc:482] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.437148 29903 raft_consensus.cc:3037] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.440048 29903 raft_consensus.cc:504] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.443281 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cd09e769be4bee95bbef9c3c550451" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:23.443981 28150 leader_election.cc:341] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:23.444207 29903 leader_election.cc:288] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:23.444914 29949 leader_election.cc:288] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.445704 27851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu pid 28736
I20240503 14:12:23.446889 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cd09e769be4bee95bbef9c3c550451" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:23.447081 28349 raft_consensus.cc:3037] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.447785 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 867) took 1815 ms. Trace:
I20240503 14:12:23.447919 29828 rpcz_store.cc:276] 0503 14:12:21.632809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.632933 (+ 124us) service_pool.cc:225] Handling call
0503 14:12:22.707354 (+1074421us) ts_tablet_manager.cc:1120] Deleting tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a
0503 14:12:22.707364 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:22.944353 (+236989us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.277164 (+332811us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.447760 (+170596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074148}
I20240503 14:12:23.448525 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 868) took 1787 ms. Trace:
I20240503 14:12:23.448676 29828 rpcz_store.cc:276] 0503 14:12:21.661178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.661248 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:23.448347 (+1787099us) ts_tablet_manager.cc:1120] Deleting tablet 10876f4512794e6aa72bda3c1da8b2d0
0503 14:12:23.448358 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.448463 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786854}
I20240503 14:12:23.449108 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 869) took 1772 ms. Trace:
I20240503 14:12:23.449232 29828 rpcz_store.cc:276] 0503 14:12:21.677030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.677088 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:23.448995 (+1771907us) ts_tablet_manager.cc:1120] Deleting tablet 281667db9a9c43d3b9d7dd9ee14b4b06
0503 14:12:23.449004 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.449092 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771653}
I20240503 14:12:23.450079 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 870) took 1772 ms. Trace:
I20240503 14:12:23.450209 29828 rpcz_store.cc:276] 0503 14:12:21.677973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.678028 (+ 55us) service_pool.cc:225] Handling call
0503 14:12:23.449967 (+1771939us) ts_tablet_manager.cc:1120] Deleting tablet 752b369f24224f2486834814a9d08495
0503 14:12:23.449976 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.450063 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771765}
I20240503 14:12:23.450608 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 871) took 1763 ms. Trace:
I20240503 14:12:23.451195 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5221fc8d6548e08e4ae2c2d466f970. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:23.472580 28290 leader_election.cc:341] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 495190fca06a40a386c8215eb2195920
I20240503 14:12:23.485054 28614 tablet_bootstrap.cc:492] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:23.485239 28614 ts_tablet_manager.cc:1367] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.532s user 0.024s sys 0.043s
I20240503 14:12:23.486100 28614 raft_consensus.cc:348] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.486305 28614 raft_consensus.cc:374] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.486359 28614 raft_consensus.cc:729] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.486552 28614 consensus_queue.cc:260] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.487046 28614 ts_tablet_manager.cc:1398] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:23.487323 28614 tablet_bootstrap.cc:492] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:23.487855 28427 consensus_peers.cc:485] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f89fca0480d4bf28160384ae0421bce. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:23.489562 28427 consensus_peers.cc:485] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0a92fc3833e49aaab8c5415de9bb685. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:23.446331 29864 raft_consensus.cc:504] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.450785 29828 rpcz_store.cc:276] 0503 14:12:21.687018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.687088 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:23.450498 (+1763410us) ts_tablet_manager.cc:1120] Deleting tablet 6874d970e4354070b11060c6b861d7bf
0503 14:12:23.450506 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.450591 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1763204}
I20240503 14:12:23.491521 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 872) took 1803 ms. Trace:
I20240503 14:12:23.491667 29828 rpcz_store.cc:276] 0503 14:12:21.687844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.687931 (+ 87us) service_pool.cc:225] Handling call
0503 14:12:23.491380 (+1803449us) ts_tablet_manager.cc:1120] Deleting tablet d952b436198141ad80fef5fb1ea1e736
0503 14:12:23.491392 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.491502 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803284}
I20240503 14:12:23.492169 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 873) took 1803 ms. Trace:
I20240503 14:12:23.492285 29828 rpcz_store.cc:276] 0503 14:12:21.689194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.689245 (+ 51us) service_pool.cc:225] Handling call
0503 14:12:23.492044 (+1802799us) ts_tablet_manager.cc:1120] Deleting tablet 60d4f5591b1f4826bc09e57094fb46bf
0503 14:12:23.492052 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.492156 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1802629}
I20240503 14:12:23.492631 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 874) took 1802 ms. Trace:
I20240503 14:12:23.492760 29828 rpcz_store.cc:276] 0503 14:12:21.689932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.689981 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:23.492540 (+1802559us) ts_tablet_manager.cc:1120] Deleting tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4
0503 14:12:23.492548 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.492617 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1802408}
I20240503 14:12:23.493139 29828 tablet_replica.cc:330] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:23.493383 29828 raft_consensus.cc:2229] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:23.493628 29828 raft_consensus.cc:2258] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:23.495131 28604 tablet_bootstrap.cc:492] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:23.495302 28604 ts_tablet_manager.cc:1367] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.437s user 0.053s sys 0.049s
I20240503 14:12:23.496120 29828 ts_tablet_manager.cc:1875] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:23.496142 28604 raft_consensus.cc:348] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.496398 28604 raft_consensus.cc:374] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.496448 28604 raft_consensus.cc:729] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.496675 28604 consensus_queue.cc:260] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.497179 28604 ts_tablet_manager.cc:1398] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.497773 28604 tablet_bootstrap.cc:492] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:23.500151 28604 tablet_bootstrap.cc:654] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.502136 28661 ts_tablet_manager.cc:1888] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:23.502256 28661 log.cc:1196] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/1c5e3649c71f4d4da710c69f85a82c25
I20240503 14:12:23.502537 28661 ts_tablet_manager.cc:1909] T 1c5e3649c71f4d4da710c69f85a82c25 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:23.522414 28026 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1066) took 1120 ms. Trace:
I20240503 14:12:23.522549 28026 rpcz_store.cc:276] 0503 14:12:22.402260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.050661 (+648401us) service_pool.cc:225] Handling call
0503 14:12:23.068457 (+ 17796us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.069386 (+ 929us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:23.069858 (+ 472us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:23.522093 (+452235us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:23.522232 (+ 139us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:23.522343 (+ 111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.461901 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.462061 (+ 160us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.462068 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:23.462071 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.462426 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.462439 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.462445 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.462786 (+ 341us) tablet.cc:699] Row locks acquired
0503 14:12:23.462790 (+ 4us) write_op.cc:260] PREPARE: finished
0503 14:12:23.462849 (+ 59us) write_op.cc:270] Start()
0503 14:12:23.462890 (+ 41us) write_op.cc:276] Timestamp: P: 1714745543462830 usec, L: 0
0503 14:12:23.462895 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.463125 (+ 230us) log.cc:842] Serialized 433 byte log entry
0503 14:12:23.519802 (+ 56677us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.520901 (+ 1099us) write_op.cc:301] APPLY: starting
0503 14:12:23.520911 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.520944 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.520979 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.521080 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.521108 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:23.521115 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:23.521167 (+ 52us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.521206 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.521221 (+ 15us) write_op.cc:454] Released schema lock
0503 14:12:23.521227 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.521274 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.521277 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":40216960,"child_traces":[["op",{"apply.queue_time_us":1072,"apply.run_cpu_time_us":415,"apply.run_wall_time_us":414,"num_ops":2,"prepare.queue_time_us":390369,"prepare.run_cpu_time_us":962,"prepare.run_wall_time_us":6168,"raft.queue_time_us":67,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":349,"replication_time_us":56890}]]}
I20240503 14:12:23.523738 28029 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1064) took 1170 ms. Trace:
I20240503 14:12:23.523855 28029 rpcz_store.cc:276] 0503 14:12:22.352802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.047423 (+694621us) service_pool.cc:225] Handling call
0503 14:12:23.070725 (+ 23302us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:23.071684 (+ 959us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:23.072225 (+ 541us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:23.523448 (+451223us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:23.523551 (+ 103us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:23.523634 (+ 83us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.466879 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.467260 (+ 381us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.467269 (+ 9us) write_op.cc:435] Acquired schema lock
0503 14:12:23.467272 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.467388 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.467402 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.467408 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.467498 (+ 90us) tablet.cc:699] Row locks acquired
0503 14:12:23.467501 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:23.467543 (+ 42us) write_op.cc:270] Start()
0503 14:12:23.467579 (+ 36us) write_op.cc:276] Timestamp: P: 1714745543467526 usec, L: 0
0503 14:12:23.467583 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.467789 (+ 206us) log.cc:842] Serialized 433 byte log entry
0503 14:12:23.520189 (+ 52400us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.521344 (+ 1155us) write_op.cc:301] APPLY: starting
0503 14:12:23.521354 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.521381 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.521401 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.521469 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.521493 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:23.521499 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:23.521539 (+ 40us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.521572 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.521583 (+ 11us) write_op.cc:454] Released schema lock
0503 14:12:23.521588 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.521630 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.521634 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":52269952,"child_traces":[["op",{"apply.queue_time_us":1125,"apply.run_cpu_time_us":322,"apply.run_wall_time_us":322,"num_ops":2,"prepare.queue_time_us":394200,"prepare.run_cpu_time_us":804,"prepare.run_wall_time_us":1437,"replication_time_us":52588,"threads_started":1}]]}
I20240503 14:12:23.527618 28600 tablet_bootstrap.cc:492] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:23.527786 28600 ts_tablet_manager.cc:1367] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.141s user 0.012s sys 0.040s
I20240503 14:12:23.528650 28600 raft_consensus.cc:348] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.528858 28600 raft_consensus.cc:374] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.528913 28600 raft_consensus.cc:729] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.529126 28600 consensus_queue.cc:260] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.529624 28600 ts_tablet_manager.cc:1398] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:23.529987 28600 tablet_bootstrap.cc:492] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:23.531211 29949 raft_consensus.cc:482] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.531366 29949 raft_consensus.cc:504] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.531540 28600 tablet_bootstrap.cc:654] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.532047 29949 leader_election.cc:288] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.508541 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 741) took 4880 ms. Trace:
I20240503 14:12:23.532349 28661 rpcz_store.cc:276] 0503 14:12:18.628467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.628541 (+ 74us) service_pool.cc:225] Handling call
0503 14:12:23.421839 (+4793298us) ts_tablet_manager.cc:1120] Deleting tablet 1c5e3649c71f4d4da710c69f85a82c25
0503 14:12:23.421851 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.492982 (+ 71131us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.502022 (+ 9040us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.508514 (+ 6492us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4793003}
I20240503 14:12:23.532832 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 742) took 4897 ms. Trace:
I20240503 14:12:23.532922 28661 rpcz_store.cc:276] 0503 14:12:18.635423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.635844 (+ 421us) service_pool.cc:225] Handling call
0503 14:12:23.532660 (+4896816us) ts_tablet_manager.cc:1120] Deleting tablet a6b468dd4102495abd4337edcfc6340a
0503 14:12:23.532673 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.532807 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4896559}
I20240503 14:12:23.534999 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 743) took 4893 ms. Trace:
I20240503 14:12:23.535131 28661 rpcz_store.cc:276] 0503 14:12:18.642004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.642073 (+ 69us) service_pool.cc:225] Handling call
0503 14:12:23.534852 (+4892779us) ts_tablet_manager.cc:1120] Deleting tablet f67c4cb980b44e9397bea70363fe1acd
0503 14:12:23.534864 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.534977 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4892523}
I20240503 14:12:23.535545 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 744) took 4867 ms. Trace:
I20240503 14:12:23.535753 29955 raft_consensus.cc:482] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.540431 29955 raft_consensus.cc:504] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.541193 29955 leader_election.cc:288] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.541445 29864 leader_election.cc:288] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.484714 28290 leader_election.cc:302] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:23.542130 29864 raft_consensus.cc:2781] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.542219 29864 raft_consensus.cc:482] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.542279 29864 raft_consensus.cc:3037] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.542248 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.542474 28213 raft_consensus.cc:2454] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:23.543085 28427 leader_election.cc:302] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:23.545660 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 60989.3us
I20240503 14:12:23.546188 28290 leader_election.cc:302] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.547317 28922 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:23.555322 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf21f6234dd64cf8a32f1599fe50746e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.555702 28491 raft_consensus.cc:2379] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:23.534639 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 1c5e3649c71f4d4da710c69f85a82c25 (table default.Table_dc507e002d97466bae378a75112560a8 [id=3a0db192d94f40c3b69f089100ad18ae]) successfully deleted
W20240503 14:12:23.558409 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a6b468dd4102495abd4337edcfc6340a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b468dd4102495abd4337edcfc6340a already in progress: creating tablet
W20240503 14:12:23.558780 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f67c4cb980b44e9397bea70363fe1acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f67c4cb980b44e9397bea70363fe1acd already in progress: creating tablet
I20240503 14:12:23.559921 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.560225 28491 raft_consensus.cc:2379] T 9b1111e7c3fe431ea653321d1dee388c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:23.540350 28661 rpcz_store.cc:276] 0503 14:12:18.668420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.669061 (+ 641us) service_pool.cc:225] Handling call
0503 14:12:23.535436 (+4866375us) ts_tablet_manager.cc:1120] Deleting tablet 0a6a58db50f849a8be729c10dbe03e76
0503 14:12:23.535446 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.535529 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4865902}
I20240503 14:12:23.562156 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 746) took 4828 ms. Trace:
I20240503 14:12:23.562305 28661 rpcz_store.cc:276] 0503 14:12:18.734169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.734233 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:23.562022 (+4827789us) ts_tablet_manager.cc:1120] Deleting tablet ac577af6460b423288a790e532b00f82
0503 14:12:23.562032 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.562136 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4827548}
I20240503 14:12:23.562544 28604 tablet_bootstrap.cc:492] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:23.562685 28604 ts_tablet_manager.cc:1367] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.031s
I20240503 14:12:23.563086 29993 raft_consensus.cc:2781] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.563982 29993 raft_consensus.cc:482] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.564204 29993 raft_consensus.cc:3037] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.564800 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1111e7c3fe431ea653321d1dee388c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.565011 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084bd4fbf604042bb33ff03c3f8cc19" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:23.565207 28488 raft_consensus.cc:3037] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.567958 28488 raft_consensus.cc:2454] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:23.564616 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a (table default.Table_79df8c9603ae456799a73fb5badd18e9 [id=23dabedfe8df4e9fa6123f2dd9b82376]) successfully deleted
I20240503 14:12:23.568284 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084bd4fbf604042bb33ff03c3f8cc19" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:23.568482 28212 raft_consensus.cc:3037] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:23.569298 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 10876f4512794e6aa72bda3c1da8b2d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10876f4512794e6aa72bda3c1da8b2d0 already in progress: creating tablet
I20240503 14:12:23.569360 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b6c998e518541248a87e4c1551ec080" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:23.569567 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 281667db9a9c43d3b9d7dd9ee14b4b06 already in progress: creating tablet
I20240503 14:12:23.569591 28491 raft_consensus.cc:2379] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
W20240503 14:12:23.569768 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 752b369f24224f2486834814a9d08495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 752b369f24224f2486834814a9d08495 already in progress: creating tablet
W20240503 14:12:23.570071 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6874d970e4354070b11060c6b861d7bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6874d970e4354070b11060c6b861d7bf already in progress: creating tablet
W20240503 14:12:23.570277 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d952b436198141ad80fef5fb1ea1e736 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d952b436198141ad80fef5fb1ea1e736 already in progress: creating tablet
W20240503 14:12:23.570431 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 60d4f5591b1f4826bc09e57094fb46bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d4f5591b1f4826bc09e57094fb46bf already in progress: creating tablet
I20240503 14:12:23.563587 28604 raft_consensus.cc:348] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.563642 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 745) took 4829 ms. Trace:
I20240503 14:12:23.571743 28604 raft_consensus.cc:374] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.545111 29864 raft_consensus.cc:504] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.574584 28353 raft_consensus.cc:3037] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.574728 29864 leader_election.cc:288] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.571746 28661 rpcz_store.cc:276] 0503 14:12:18.733995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.734240 (+ 245us) service_pool.cc:225] Handling call
0503 14:12:23.562685 (+4828445us) ts_tablet_manager.cc:1120] Deleting tablet 7927f3e0fb214441aaf52352b6956ef6
0503 14:12:23.562694 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.563623 (+ 929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4828083}
I20240503 14:12:23.575244 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 747) took 4810 ms. Trace:
I20240503 14:12:23.575333 28661 rpcz_store.cc:276] 0503 14:12:18.764374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:18.764426 (+ 52us) service_pool.cc:225] Handling call
0503 14:12:23.575095 (+4810669us) ts_tablet_manager.cc:1120] Deleting tablet e69e01bdbae349d0a61a295b6f6352f0
0503 14:12:23.575107 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.575222 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4810412}
I20240503 14:12:23.575713 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 756) took 4513 ms. Trace:
I20240503 14:12:23.575773 28661 rpcz_store.cc:276] 0503 14:12:19.061993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.062046 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:23.575601 (+4513555us) ts_tablet_manager.cc:1120] Deleting tablet cc004243d5a947e3a8852605b0414c95
0503 14:12:23.575611 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.575698 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4513278}
I20240503 14:12:23.575985 28661 tablet_replica.cc:330] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:23.576203 28661 raft_consensus.cc:2229] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:23.576508 28661 pending_rounds.cc:62] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:23.576560 28661 pending_rounds.cc:68] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597732861992960 op_type: NO_OP noop_request { }
I20240503 14:12:23.576691 28661 raft_consensus.cc:2866] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:23.576773 28661 raft_consensus.cc:2258] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:23.577431 28661 ts_tablet_manager.cc:1875] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:23.577497 29996 raft_consensus.cc:482] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.577736 29996 raft_consensus.cc:504] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.578429 29996 leader_election.cc:288] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.571817 28604 raft_consensus.cc:729] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.580174 28604 consensus_queue.cc:260] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.582334 29903 consensus_queue.cc:1035] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:23.571694 29993 raft_consensus.cc:504] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.584184 28349 raft_consensus.cc:2454] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:23.584602 28150 leader_election.cc:302] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.584414 28604 ts_tablet_manager.cc:1398] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 14:12:23.585057 29903 raft_consensus.cc:2781] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.585109 28604 tablet_bootstrap.cc:492] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:23.585192 29903 raft_consensus.cc:686] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:23.585458 29903 consensus_queue.cc:237] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.585968 28614 tablet_bootstrap.cc:654] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.588753 29949 raft_consensus.cc:2726] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:23.594305 29993 leader_election.cc:288] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.601601 28604 tablet_bootstrap.cc:654] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.606192 28614 tablet_bootstrap.cc:492] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:23.606379 28614 ts_tablet_manager.cc:1367] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.001s
I20240503 14:12:23.607124 28614 raft_consensus.cc:348] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.607347 28614 raft_consensus.cc:374] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.607399 28614 raft_consensus.cc:729] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.607656 28614 consensus_queue.cc:260] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.608382 28614 ts_tablet_manager.cc:1398] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:23.608662 28614 tablet_bootstrap.cc:492] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:23.609776 28661 ts_tablet_manager.cc:1888] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:23.610293 28614 tablet_bootstrap.cc:654] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.620226 28661 log.cc:1196] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/5dd8b2adaeeb4c3abfbfe68cc005e03a
I20240503 14:12:23.620844 28661 ts_tablet_manager.cc:1909] T 5dd8b2adaeeb4c3abfbfe68cc005e03a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:23.621903 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 755) took 4560 ms. Trace:
I20240503 14:12:23.622103 28661 rpcz_store.cc:276] 0503 14:12:19.061877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.069065 (+ 7188us) service_pool.cc:225] Handling call
0503 14:12:23.575943 (+4506878us) ts_tablet_manager.cc:1120] Deleting tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a
0503 14:12:23.575951 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.598680 (+ 22729us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.609646 (+ 10966us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.621872 (+ 12226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4506549}
I20240503 14:12:23.622640 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 758) took 4557 ms. Trace:
I20240503 14:12:23.622788 28661 rpcz_store.cc:276] 0503 14:12:19.065224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.069553 (+ 4329us) service_pool.cc:225] Handling call
0503 14:12:23.622494 (+4552941us) ts_tablet_manager.cc:1120] Deleting tablet cace51f8e5ff49f596c94f3698fdf9ef
0503 14:12:23.622504 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.622622 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4552734}
I20240503 14:12:23.623203 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 757) took 4558 ms. Trace:
I20240503 14:12:23.623334 28661 rpcz_store.cc:276] 0503 14:12:19.064838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.064901 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:23.623088 (+4558187us) ts_tablet_manager.cc:1120] Deleting tablet 6809b59e87644316b9a450d6b670da2b
0503 14:12:23.623098 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.623185 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4546610}
I20240503 14:12:23.623720 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 759) took 4538 ms. Trace:
I20240503 14:12:23.623847 28661 rpcz_store.cc:276] 0503 14:12:19.085203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.089955 (+ 4752us) service_pool.cc:225] Handling call
0503 14:12:23.623615 (+4533660us) ts_tablet_manager.cc:1120] Deleting tablet bf8d2474104a41a989cd7bd4480367f5
0503 14:12:23.623623 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.623705 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4533334}
I20240503 14:12:23.624228 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 761) took 4521 ms. Trace:
I20240503 14:12:23.624349 28661 rpcz_store.cc:276] 0503 14:12:19.102922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.103074 (+ 152us) service_pool.cc:225] Handling call
0503 14:12:23.624131 (+4521057us) ts_tablet_manager.cc:1120] Deleting tablet 239f14be29e54cd29bb5e96622aacd55
0503 14:12:23.624138 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.624212 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4520787}
I20240503 14:12:23.624975 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 770) took 4214 ms. Trace:
I20240503 14:12:23.625099 28661 rpcz_store.cc:276] 0503 14:12:19.410334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.410757 (+ 423us) service_pool.cc:225] Handling call
0503 14:12:23.624877 (+4214120us) ts_tablet_manager.cc:1120] Deleting tablet 08cc84cf703048148abfe03b7dee12ce
0503 14:12:23.624886 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.624959 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4213839}
I20240503 14:12:23.625404 28661 tablet_replica.cc:330] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:23.625643 28661 raft_consensus.cc:2229] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:23.625865 28661 raft_consensus.cc:2258] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:23.626456 28661 ts_tablet_manager.cc:1875] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:23.631412 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbcae22ea124edaa14c9409f3052714" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.632550 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c61b9232ebd4431a3d4202ad971d540" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:23.632628 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbcae22ea124edaa14c9409f3052714" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.632795 28488 raft_consensus.cc:3037] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.632875 28213 raft_consensus.cc:2454] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:23.635861 28488 raft_consensus.cc:2454] T 3c61b9232ebd4431a3d4202ad971d540 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:23.636230 28604 tablet_bootstrap.cc:492] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:23.636387 28604 ts_tablet_manager.cc:1367] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240503 14:12:23.637266 28604 raft_consensus.cc:348] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.637470 28604 raft_consensus.cc:374] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.637527 28604 raft_consensus.cc:729] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.637737 28604 consensus_queue.cc:260] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.638275 28604 ts_tablet_manager.cc:1398] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:23.638556 28604 tablet_bootstrap.cc:492] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:23.640949 28032 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1067) took 1231 ms. Trace:
W20240503 14:12:23.641608 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d348da459634561985c4ac1710c0bb2 consensus state has no leader
I20240503 14:12:23.641101 28032 rpcz_store.cc:276] 0503 14:12:22.409267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.378544 (+969277us) service_pool.cc:225] Handling call
0503 14:12:23.379702 (+ 1158us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:23.382038 (+ 2336us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:23.640429 (+258391us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:23.640625 (+ 196us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:23.640756 (+ 131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.468423 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.468531 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.468538 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:23.468541 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.468656 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.468667 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.468673 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:23.468756 (+ 83us) tablet.cc:699] Row locks acquired
0503 14:12:23.468758 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:23.468792 (+ 34us) write_op.cc:270] Start()
0503 14:12:23.468820 (+ 28us) write_op.cc:276] Timestamp: P: 1714745543468778 usec, L: 0
0503 14:12:23.468824 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.468978 (+ 154us) log.cc:842] Serialized 433 byte log entry
0503 14:12:23.639604 (+170626us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.639803 (+ 199us) write_op.cc:301] APPLY: starting
0503 14:12:23.639822 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.639864 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.639896 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.640055 (+ 159us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.640116 (+ 61us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:23.640124 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:23.640227 (+ 103us) log.cc:842] Serialized 29 byte log entry
0503 14:12:23.640310 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.640330 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:23.640337 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.640566 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.640570 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":685,"apply.run_wall_time_us":1042,"num_ops":2,"prepare.queue_time_us":85822,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":957,"replication_time_us":170749}]]}
I20240503 14:12:23.645874 29949 raft_consensus.cc:482] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.646056 29949 raft_consensus.cc:504] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.646131 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c61b9232ebd4431a3d4202ad971d540" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:23.646389 28213 raft_consensus.cc:3037] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.646776 29949 leader_election.cc:288] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.647272 29949 raft_consensus.cc:482] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.647406 29949 raft_consensus.cc:504] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.647991 29949 leader_election.cc:288] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.648311 29949 raft_consensus.cc:482] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.648427 29949 raft_consensus.cc:504] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.648983 29949 leader_election.cc:288] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:23.643986 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 97063us
I20240503 14:12:23.652421 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d0f835c02d5f4ffea64369bb8db62de9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:23.653422 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4489c34fb7d34e72892ce61b61a5fe17" } new_table_name: "default.Table_8ae143ed239d4c81bd80ed397373c7da" modify_external_catalogs: true
I20240503 14:12:23.656466 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:23.656633 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d28d3c2ea472191ad21355248a2fc" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.656831 28209 raft_consensus.cc:2454] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:23.656700 28216 raft_consensus.cc:3037] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.657891 29903 raft_consensus.cc:482] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.658035 29903 raft_consensus.cc:504] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.658183 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_40fb6c7d8a074d12962a5f9b17b3c14f" } new_table_name: "default.Table_017d6581de514190afe69b1b1585139b" modify_external_catalogs: true
I20240503 14:12:23.658672 29903 leader_election.cc:288] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:23.659533 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe3affd0d144b1fb0078e7690b9b36d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:23.660151 28150 leader_election.cc:341] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:23.660523 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8e709ec40c1b4306944f2be34700e617" } modify_external_catalogs: true
I20240503 14:12:23.660753 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8e709ec40c1b4306944f2be34700e617" } modify_external_catalogs: true
W20240503 14:12:23.649747 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0a6a58db50f849a8be729c10dbe03e76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6a58db50f849a8be729c10dbe03e76 already in progress: creating tablet
W20240503 14:12:23.666692 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0f835c02d5f4ffea64369bb8db62de9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:23.666967 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ac577af6460b423288a790e532b00f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac577af6460b423288a790e532b00f82 already in progress: creating tablet
W20240503 14:12:23.667304 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7927f3e0fb214441aaf52352b6956ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7927f3e0fb214441aaf52352b6956ef6 already in progress: creating tablet
W20240503 14:12:23.667593 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e69e01bdbae349d0a61a295b6f6352f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e69e01bdbae349d0a61a295b6f6352f0 already in progress: creating tablet
W20240503 14:12:23.667856 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet cc004243d5a947e3a8852605b0414c95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc004243d5a947e3a8852605b0414c95 already in progress: creating tablet
I20240503 14:12:23.668058 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a (table default.Table_79df8c9603ae456799a73fb5badd18e9 [id=23dabedfe8df4e9fa6123f2dd9b82376]) successfully deleted
W20240503 14:12:23.668392 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet cace51f8e5ff49f596c94f3698fdf9ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cace51f8e5ff49f596c94f3698fdf9ef already in progress: creating tablet
W20240503 14:12:23.668617 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6809b59e87644316b9a450d6b670da2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6809b59e87644316b9a450d6b670da2b already in progress: creating tablet
W20240503 14:12:23.668941 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet bf8d2474104a41a989cd7bd4480367f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8d2474104a41a989cd7bd4480367f5 already in progress: creating tablet
W20240503 14:12:23.669220 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 239f14be29e54cd29bb5e96622aacd55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 239f14be29e54cd29bb5e96622aacd55 already in progress: creating tablet
W20240503 14:12:23.669528 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 08cc84cf703048148abfe03b7dee12ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cc84cf703048148abfe03b7dee12ce already in progress: creating tablet
I20240503 14:12:23.680312 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9b093db5f6d4c709a15d7dc1ca6059b" } modify_external_catalogs: true
I20240503 14:12:23.680716 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a9b093db5f6d4c709a15d7dc1ca6059b" } modify_external_catalogs: true
I20240503 14:12:23.681782 28604 tablet_bootstrap.cc:654] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.661877 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe3affd0d144b1fb0078e7690b9b36d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.683259 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_68199a1d18244f54bb0b7b0256a58e2a" } modify_external_catalogs: true
W20240503 14:12:23.683485 28150 leader_election.cc:341] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:23.683516 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_68199a1d18244f54bb0b7b0256a58e2a" } modify_external_catalogs: true
I20240503 14:12:23.683622 28150 leader_election.cc:302] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.684127 29903 raft_consensus.cc:2726] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:23.686591 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7549b2ca31b04247bf3a2202e7585817 consensus state has no leader
I20240503 14:12:23.687103 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet e21a5dafa82f4660927d96c779a23478 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:23.689465 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d8ddcdb3c764dc59e9291725046d839 consensus state has no leader
I20240503 14:12:23.689324 28661 ts_tablet_manager.cc:1888] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:23.690429 28661 log.cc:1196] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6c7c5dfac02d4dfe932996d2a37fe031
I20240503 14:12:23.691435 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e141d550a34a4f838e7db3335d0ea5ce" } modify_external_catalogs: true
I20240503 14:12:23.691813 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e141d550a34a4f838e7db3335d0ea5ce" } modify_external_catalogs: true
I20240503 14:12:23.691955 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5fb312c322b34390a3823a7d55e21ebe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.692647 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fb312c322b34390a3823a7d55e21ebe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.693289 28661 ts_tablet_manager.cc:1909] T 6c7c5dfac02d4dfe932996d2a37fe031 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:23.694551 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_905413b2b2024c9685068fe4c869f748" } modify_external_catalogs: true
I20240503 14:12:23.694801 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_905413b2b2024c9685068fe4c869f748" } modify_external_catalogs: true
W20240503 14:12:23.693385 28427 consensus_peers.cc:485] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95c97a43bd714da68c1a77408f3d1cd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:23.708067 28427 leader_election.cc:302] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:23.708391 29955 raft_consensus.cc:2781] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.708482 29955 raft_consensus.cc:482] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.708549 29955 raft_consensus.cc:3037] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.711337 29955 raft_consensus.cc:504] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.712157 29955 leader_election.cc:288] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.715946 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:23.716274 28349 raft_consensus.cc:2454] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:23.716925 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d28d3c2ea472191ad21355248a2fc" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:23.717124 28349 raft_consensus.cc:2454] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:23.717551 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:23.717723 28349 raft_consensus.cc:3037] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.720974 28349 raft_consensus.cc:2454] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:23.724740 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5297664a4d943c28adfb6ac4f061680" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:23.725016 28488 raft_consensus.cc:3037] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.725937 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd28bc07e4ccd9097f6dee471c5c6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.726192 28491 raft_consensus.cc:2454] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:23.727532 28290 leader_election.cc:341] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: bf21f6234dd64cf8a32f1599fe50746e
I20240503 14:12:23.731655 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.734612 28212 raft_consensus.cc:2454] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:23.735018 28213 raft_consensus.cc:2454] T 3c61b9232ebd4431a3d4202ad971d540 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:23.735592 28600 tablet_bootstrap.cc:492] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:23.735754 28600 ts_tablet_manager.cc:1367] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.206s user 0.006s sys 0.013s
I20240503 14:12:23.702754 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 771) took 4253 ms. Trace:
I20240503 14:12:23.736527 28661 rpcz_store.cc:276] 0503 14:12:19.448639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.448702 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:23.625362 (+4176660us) ts_tablet_manager.cc:1120] Deleting tablet 6c7c5dfac02d4dfe932996d2a37fe031
0503 14:12:23.625370 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.633922 (+ 8552us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.688445 (+ 54523us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.702521 (+ 14076us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4176398}
I20240503 14:12:23.736593 28600 raft_consensus.cc:348] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.736771 28600 raft_consensus.cc:374] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.736820 28600 raft_consensus.cc:729] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.737030 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 772) took 4252 ms. Trace:
I20240503 14:12:23.737035 28600 consensus_queue.cc:260] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.737105 28661 rpcz_store.cc:276] 0503 14:12:19.484384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.484472 (+ 88us) service_pool.cc:225] Handling call
0503 14:12:23.736863 (+4252391us) ts_tablet_manager.cc:1120] Deleting tablet 2c1db04ab92c4af0847730a47635a697
0503 14:12:23.736876 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.737005 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4252071}
I20240503 14:12:23.737457 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 773) took 4236 ms. Trace:
I20240503 14:12:23.737587 28216 raft_consensus.cc:2454] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:23.737529 28661 rpcz_store.cc:276] 0503 14:12:19.500836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.500923 (+ 87us) service_pool.cc:225] Handling call
0503 14:12:23.737337 (+4236414us) ts_tablet_manager.cc:1120] Deleting tablet 37364d5facc241f4bbccc979a50a0ca8
0503 14:12:23.737347 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.737440 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4201450}
I20240503 14:12:23.737624 28600 ts_tablet_manager.cc:1398] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.737987 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 774) took 4214 ms. Trace:
I20240503 14:12:23.738029 28600 tablet_bootstrap.cc:492] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:23.738068 28661 rpcz_store.cc:276] 0503 14:12:19.523831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.524267 (+ 436us) service_pool.cc:225] Handling call
0503 14:12:23.737862 (+4213595us) ts_tablet_manager.cc:1120] Deleting tablet 03700c1efc08471cb67dfe3a8016a740
0503 14:12:23.737871 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.737968 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4201904}
I20240503 14:12:23.740983 29903 raft_consensus.cc:482] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.741180 29903 raft_consensus.cc:504] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.741860 29903 leader_election.cc:288] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:23.743940 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43823da09bb49be8f5d1a1149a4b11d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:23.744218 28349 raft_consensus.cc:2454] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:23.744658 28150 leader_election.cc:302] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:23.745112 29903 raft_consensus.cc:2781] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.745214 29903 raft_consensus.cc:482] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.745272 29903 raft_consensus.cc:3037] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.745710 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd28bc07e4ccd9097f6dee471c5c6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:23.745939 28207 raft_consensus.cc:2454] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:23.748853 29903 raft_consensus.cc:504] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.755816 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43823da09bb49be8f5d1a1149a4b11d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:23.755956 28600 tablet_bootstrap.cc:654] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.756072 28493 raft_consensus.cc:3037] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.757400 29903 leader_election.cc:288] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:23.760722 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43823da09bb49be8f5d1a1149a4b11d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:23.761051 28353 raft_consensus.cc:3037] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.771957 29828 ts_tablet_manager.cc:1888] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:23.772253 29828 log.cc:1196] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/77f0081cc2cb48aaa368741109032fd1
I20240503 14:12:23.772791 29828 ts_tablet_manager.cc:1909] T 77f0081cc2cb48aaa368741109032fd1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:23.773079 28353 raft_consensus.cc:2454] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:23.773672 28150 leader_election.cc:302] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:23.782320 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_85dbdd19a2514c63bc5ee54794385f56" } new_table_name: "default.Table_0622233baec140adb2da5612db055913" modify_external_catalogs: true
I20240503 14:12:23.742671 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43823da09bb49be8f5d1a1149a4b11d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.789211 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410e8b6b511403ea609b1353855937b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.789498 28486 raft_consensus.cc:2379] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:23.791235 28030 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a248a48a77f94703a1440b059015d7dd
I20240503 14:12:23.796774 30005 raft_consensus.cc:2781] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.799078 28600 tablet_bootstrap.cc:492] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:23.799472 28600 ts_tablet_manager.cc:1367] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.003s
I20240503 14:12:23.800382 28600 raft_consensus.cc:348] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.800787 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 777) took 4205 ms. Trace:
I20240503 14:12:23.800938 28600 raft_consensus.cc:374] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.801046 28600 raft_consensus.cc:729] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.801345 28600 consensus_queue.cc:260] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.801666 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 875) took 2099 ms. Trace:
I20240503 14:12:23.801980 28600 ts_tablet_manager.cc:1398] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.801824 29828 rpcz_store.cc:276] 0503 14:12:21.701815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.701875 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:23.493093 (+1791218us) ts_tablet_manager.cc:1120] Deleting tablet 77f0081cc2cb48aaa368741109032fd1
0503 14:12:23.493101 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.570418 (+ 77317us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.771808 (+201390us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.801633 (+ 29825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1791007}
I20240503 14:12:23.802311 28600 tablet_bootstrap.cc:492] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:23.802460 29828 tablet_replica.cc:330] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:23.807396 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d039e7aa896743fe87bb9365dfe8a704" } new_table_name: "default.Table_bde2d7996f8f41e991c8e0f5b7b166c6" modify_external_catalogs: true
I20240503 14:12:23.801088 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6202333af943cbbc0d09553c1c193c" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:23.810801 28486 raft_consensus.cc:2379] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:23.816979 30005 raft_consensus.cc:686] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:23.818189 28488 raft_consensus.cc:2454] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:23.819128 30010 consensus_queue.cc:1035] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:23.819977 30005 consensus_queue.cc:237] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.822692 28600 tablet_bootstrap.cc:654] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.824626 28493 raft_consensus.cc:2454] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:23.833767 28604 tablet_bootstrap.cc:492] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:23.833976 28604 ts_tablet_manager.cc:1367] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.195s user 0.017s sys 0.031s
I20240503 14:12:23.834796 28604 raft_consensus.cc:348] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.835031 28604 raft_consensus.cc:374] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.835100 28604 raft_consensus.cc:729] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.838809 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d28d3c2ea472191ad21355248a2fc" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:23.839041 28207 raft_consensus.cc:3037] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:23.839382 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 111587us
I20240503 14:12:23.841941 28207 raft_consensus.cc:2454] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:23.845113 28290 leader_election.cc:341] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 7b6c998e518541248a87e4c1551ec080
I20240503 14:12:23.845355 28600 tablet_bootstrap.cc:492] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:23.800887 28661 rpcz_store.cc:276] 0503 14:12:19.594924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.595315 (+ 391us) service_pool.cc:225] Handling call
0503 14:12:23.738280 (+4142965us) ts_tablet_manager.cc:1120] Deleting tablet 6de9e159d2ba47039dd6bd207a9cddeb
0503 14:12:23.738289 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.800756 (+ 62467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4142742}
I20240503 14:12:23.846747 28661 tablet_replica.cc:330] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:23.847025 28661 raft_consensus.cc:2229] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:23.847268 28661 raft_consensus.cc:2258] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:23.847957 28661 ts_tablet_manager.cc:1875] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:23.847316 28600 ts_tablet_manager.cc:1367] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240503 14:12:23.848588 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6c7c5dfac02d4dfe932996d2a37fe031 (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
I20240503 14:12:23.849067 28290 leader_election.cc:302] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
W20240503 14:12:23.849174 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2c1db04ab92c4af0847730a47635a697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c1db04ab92c4af0847730a47635a697 already in progress: creating tablet
I20240503 14:12:23.849352 28600 raft_consensus.cc:348] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:23.849669 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 37364d5facc241f4bbccc979a50a0ca8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37364d5facc241f4bbccc979a50a0ca8 already in progress: creating tablet
W20240503 14:12:23.850078 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 03700c1efc08471cb67dfe3a8016a740 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03700c1efc08471cb67dfe3a8016a740 already in progress: creating tablet
I20240503 14:12:23.852123 29949 raft_consensus.cc:2781] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.852257 29949 raft_consensus.cc:482] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.852327 29949 raft_consensus.cc:3037] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.853811 28614 tablet_bootstrap.cc:492] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:23.853960 28614 ts_tablet_manager.cc:1367] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.245s user 0.022s sys 0.050s
I20240503 14:12:23.854786 28614 raft_consensus.cc:348] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.854982 28614 raft_consensus.cc:374] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.855031 28614 raft_consensus.cc:729] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.855093 29949 raft_consensus.cc:504] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.855217 28614 consensus_queue.cc:260] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.855770 28614 ts_tablet_manager.cc:1398] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:23.856133 28614 tablet_bootstrap.cc:492] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:23.856434 29949 leader_election.cc:288] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:23.856876 28290 leader_election.cc:341] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: c5297664a4d943c28adfb6ac4f061680
W20240503 14:12:23.857339 28290 leader_election.cc:341] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 8410e8b6b511403ea609b1353855937b
I20240503 14:12:23.857605 28614 tablet_bootstrap.cc:654] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.858481 28600 raft_consensus.cc:374] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.858659 28600 raft_consensus.cc:729] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.858873 29996 consensus_queue.cc:1035] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:23.859021 28600 consensus_queue.cc:260] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.859695 28600 ts_tablet_manager.cc:1398] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 14:12:23.864971 28604 consensus_queue.cc:260] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.864820 28024 catalog_manager.cc:5484] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:23.865381 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
W20240503 14:12:23.868358 28427 consensus_peers.cc:485] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f89fca0480d4bf28160384ae0421bce. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:23.870105 28604 ts_tablet_manager.cc:1398] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20240503 14:12:23.870524 28604 tablet_bootstrap.cc:492] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:23.857578 28290 leader_election.cc:302] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:23.881600 28290 leader_election.cc:302] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:23.883044 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_023451ab297b451da33176a3d13fe295"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.883764 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_023451ab297b451da33176a3d13fe295 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.862677 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 587240644403401f8d226942b962f683 (DEFAULT_TABLE table=default.Table_715482091e944a8abe222f740b5be62b [id=37f5645a541d4c55b96766ebd210ffb8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:23.885506 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cce4c617ee694c36b3e63d51d6e0f209 consensus state has no leader
I20240503 14:12:23.885679 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587240644403401f8d226942b962f683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.886039 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1ef3157c088942e4ab395d7e11d0a29d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.886660 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ef3157c088942e4ab395d7e11d0a29d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.877347 28179 tablet_service.cc:1467] Processing CreateTablet for tablet f4f74d225a13429395676d1d91253a79 (DEFAULT_TABLE table=default.Table_30c6122e36244089a4eba499c8881c3f [id=4b1179f88f3e44269f1b582504e17d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.888028 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 1132014802ed4c3682769ba86312eed2
I20240503 14:12:23.888494 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f74d225a13429395676d1d91253a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.888839 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet a31555fc800142bd849dbd572a194002 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:23.889544 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c2cb53287e4e4b3781e13698a10d0f43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:23.890170 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2cb53287e4e4b3781e13698a10d0f43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.880874 28193 tablet_service.cc:1467] Processing CreateTablet for tablet d21bdd6490004300bca7a5a7fb24cd07 (DEFAULT_TABLE table=default.Table_e141d550a34a4f838e7db3335d0ea5ce [id=8b0b765740a248efa85b8ebc34f33652]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.892459 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21bdd6490004300bca7a5a7fb24cd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.898113 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 1132014802ed4c3682769ba86312eed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:23.883270 30015 raft_consensus.cc:2781] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.911430 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1bc119df7d4694bbe926611ef73245 (DEFAULT_TABLE table=default.Table_2841ebe2e868418690799c3ccd7901df [id=b292d157f70c41d7a3220b88a7cd83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.912153 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1bc119df7d4694bbe926611ef73245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.913383 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 11e14e8b5f4046029af4673a277112d9
I20240503 14:12:23.914273 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c9669afd79724f9e9f03dbcd90efab74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:23.914980 28600 tablet_bootstrap.cc:492] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:23.915104 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9669afd79724f9e9f03dbcd90efab74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:23.915410 29949 raft_consensus.cc:2781] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.915555 29949 raft_consensus.cc:686] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:23.915827 29949 consensus_queue.cc:237] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.917632 30015 raft_consensus.cc:686] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:23.917971 30015 consensus_queue.cc:237] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.918582 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2b9a6aca21d746eb8f295ed5ea44c9d9
I20240503 14:12:23.919461 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet abb9279e91124fe6bfbdc8856eccfed8 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:23.924153 28427 consensus_peers.cc:485] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0a92fc3833e49aaab8c5415de9bb685. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:23.925854 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 50812.9us
I20240503 14:12:23.928707 28427 leader_election.cc:302] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:23.929139 29955 raft_consensus.cc:2781] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.929970 29955 raft_consensus.cc:686] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:23.930339 29955 consensus_queue.cc:237] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.934249 28427 leader_election.cc:302] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:23.940063 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 58492.3us
I20240503 14:12:23.940474 28290 leader_election.cc:302] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:23.940912 29864 raft_consensus.cc:2781] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.941007 29864 raft_consensus.cc:482] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.941068 29864 raft_consensus.cc:3037] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:23.942710 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6de9e159d2ba47039dd6bd207a9cddeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6de9e159d2ba47039dd6bd207a9cddeb already in progress: creating tablet
I20240503 14:12:23.943922 29864 raft_consensus.cc:504] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.944996 29864 leader_election.cc:288] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:23.921180 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3896fbc6534762bfabce5050a921cc (DEFAULT_TABLE table=default.Table_60e99647c73642ceb431834009fa427e [id=5b324e27d45240bf926e6b0e764362be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.946189 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3896fbc6534762bfabce5050a921cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.947309 28600 tablet_bootstrap.cc:654] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.951186 28290 leader_election.cc:302] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.951617 29864 raft_consensus.cc:3037] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.921211 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:23.954735 29864 raft_consensus.cc:2726] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:23.925397 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e14e8b5f4046029af4673a277112d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:23.957312 28290 leader_election.cc:302] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.957749 29864 raft_consensus.cc:2726] T 9b1111e7c3fe431ea653321d1dee388c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:23.964562 28290 leader_election.cc:302] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.965013 29864 raft_consensus.cc:3037] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.966298 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d21bdd6490004300bca7a5a7fb24cd07
I20240503 14:12:23.967032 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 57ab077988364bd687e54f90c3c265d5 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:23.967703 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_12b9dd4031b04dccb4e6df3318d79588" } new_table_name: "default.Table_a5de24e133584200af645a728b1346e1" modify_external_catalogs: true
I20240503 14:12:23.967797 29864 raft_consensus.cc:2726] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:23.973654 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 1132014802ed4c3682769ba86312eed2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1132014802ed4c3682769ba86312eed2 already in progress: creating tablet
W20240503 14:12:23.974238 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 already in progress: creating tablet
W20240503 14:12:23.974757 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 11e14e8b5f4046029af4673a277112d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11e14e8b5f4046029af4673a277112d9 already in progress: creating tablet
I20240503 14:12:23.975325 28033 catalog_manager.cc:5484] T 9b1111e7c3fe431ea653321d1dee388c P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:23.977352 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet d21bdd6490004300bca7a5a7fb24cd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:23.978395 28290 leader_election.cc:341] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:23.979247 28290 leader_election.cc:302] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.979810 28290 leader_election.cc:302] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:23.980295 28290 leader_election.cc:302] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:23.980584 29864 raft_consensus.cc:3037] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.981410 30015 raft_consensus.cc:2726] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:23.981700 30015 raft_consensus.cc:2781] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.981865 30015 raft_consensus.cc:686] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:23.982220 30015 consensus_queue.cc:237] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.984066 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 530f24322a454f7bb04e818ca865a369 (DEFAULT_TABLE table=default.Table_ef2d2d784a834891be22d51c1d9f7ae2 [id=a0701af20254420f952b58c1ebe298b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.984633 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530f24322a454f7bb04e818ca865a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.986132 28661 ts_tablet_manager.cc:1888] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:23.988071 28462 tablet_service.cc:1467] Processing CreateTablet for tablet d21bdd6490004300bca7a5a7fb24cd07 (DEFAULT_TABLE table=default.Table_e141d550a34a4f838e7db3335d0ea5ce [id=8b0b765740a248efa85b8ebc34f33652]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.988242 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:23.990379 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 530f24322a454f7bb04e818ca865a369 (DEFAULT_TABLE table=default.Table_ef2d2d784a834891be22d51c1d9f7ae2 [id=a0701af20254420f952b58c1ebe298b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.991624 29996 raft_consensus.cc:2781] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.991847 29996 raft_consensus.cc:686] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:23.993257 29996 consensus_queue.cc:237] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.994505 28464 tablet_service.cc:1467] Processing CreateTablet for tablet f4f74d225a13429395676d1d91253a79 (DEFAULT_TABLE table=default.Table_30c6122e36244089a4eba499c8881c3f [id=4b1179f88f3e44269f1b582504e17d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.996474 28604 tablet_bootstrap.cc:654] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.997619 29864 raft_consensus.cc:2726] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:23.998567 30010 raft_consensus.cc:482] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.998646 29828 raft_consensus.cc:2229] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:23.998736 30010 raft_consensus.cc:504] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:23.998852 29828 raft_consensus.cc:2258] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:23.999442 30010 leader_election.cc:288] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:24.000185 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920be6d87c4440d0bf7453c86cfe0b5c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:24.000188 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920be6d87c4440d0bf7453c86cfe0b5c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:24.000406 28353 raft_consensus.cc:2454] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
W20240503 14:12:24.000856 28150 leader_election.cc:341] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.001305 28150 leader_election.cc:302] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:24.001619 30010 raft_consensus.cc:2781] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:24.001699 30010 raft_consensus.cc:482] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:24.001761 30010 raft_consensus.cc:3037] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.001911 29811 consensus_queue.cc:1035] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.005687 30010 raft_consensus.cc:504] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.007256 29828 ts_tablet_manager.cc:1875] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.008445 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920be6d87c4440d0bf7453c86cfe0b5c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:24.009333 28150 leader_election.cc:341] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.010596 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920be6d87c4440d0bf7453c86cfe0b5c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:24.010804 28353 raft_consensus.cc:3037] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.013625 28353 raft_consensus.cc:2454] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:24.014438 28150 leader_election.cc:302] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:24.014891 30010 leader_election.cc:288] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:24.015583 30010 raft_consensus.cc:2781] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.016043 30010 raft_consensus.cc:686] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:24.016686 30010 consensus_queue.cc:237] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:24.027249 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d21bdd6490004300bca7a5a7fb24cd07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d21bdd6490004300bca7a5a7fb24cd07 already in progress: creating tablet
I20240503 14:12:24.029049 28614 tablet_bootstrap.cc:492] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.029311 28614 ts_tablet_manager.cc:1367] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.173s user 0.002s sys 0.003s
W20240503 14:12:24.034150 28427 consensus_peers.cc:485] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed4dc732bcb44222b4b15e5d77defe33. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.037288 28600 tablet_bootstrap.cc:492] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.037571 28600 ts_tablet_manager.cc:1367] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.002s
I20240503 14:12:24.038404 28600 raft_consensus.cc:348] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.038758 28600 raft_consensus.cc:374] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.038856 28600 raft_consensus.cc:729] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.039158 28600 consensus_queue.cc:260] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.039973 28600 ts_tablet_manager.cc:1398] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:24.044446 28614 raft_consensus.cc:348] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.044809 28614 raft_consensus.cc:374] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.044907 28614 raft_consensus.cc:729] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.045194 28614 consensus_queue.cc:260] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.045882 28614 ts_tablet_manager.cc:1398] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20240503 14:12:24.046970 28614 tablet_bootstrap.cc:492] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.048944 28600 tablet_bootstrap.cc:492] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.043542 28197 tablet_service.cc:1467] Processing CreateTablet for tablet e6871d6c3610402fa13042ff86a14e11 (DEFAULT_TABLE table=default.Table_e732ed678eff478eb9c213eb0abc07bc [id=a4cd100ac6f94d1ebc017ca89bbbf852]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.050585 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6871d6c3610402fa13042ff86a14e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.051657 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 1132014802ed4c3682769ba86312eed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:24.051908 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1bc119df7d4694bbe926611ef73245 (DEFAULT_TABLE table=default.Table_2841ebe2e868418690799c3ccd7901df [id=b292d157f70c41d7a3220b88a7cd83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.052390 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21bdd6490004300bca7a5a7fb24cd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.056356 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3896fbc6534762bfabce5050a921cc (DEFAULT_TABLE table=default.Table_60e99647c73642ceb431834009fa427e [id=5b324e27d45240bf926e6b0e764362be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.056963 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3896fbc6534762bfabce5050a921cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.060252 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 587240644403401f8d226942b962f683 (DEFAULT_TABLE table=default.Table_715482091e944a8abe222f740b5be62b [id=37f5645a541d4c55b96766ebd210ffb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.060804 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587240644403401f8d226942b962f683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.063663 28457 tablet_service.cc:1467] Processing CreateTablet for tablet e6871d6c3610402fa13042ff86a14e11 (DEFAULT_TABLE table=default.Table_e732ed678eff478eb9c213eb0abc07bc [id=a4cd100ac6f94d1ebc017ca89bbbf852]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.064298 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6871d6c3610402fa13042ff86a14e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.066696 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e14e8b5f4046029af4673a277112d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:24.066910 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet d21bdd6490004300bca7a5a7fb24cd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:24.071784 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1bc119df7d4694bbe926611ef73245. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:24.046368 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 66084.8us
I20240503 14:12:24.051903 29828 ts_tablet_manager.cc:1888] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:24.076285 29828 log.cc:1196] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/dda39f0323ec47799d3d44c96f3ba605
I20240503 14:12:24.076673 29828 ts_tablet_manager.cc:1909] T dda39f0323ec47799d3d44c96f3ba605 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:24.077070 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 882) took 2202 ms. Trace:
I20240503 14:12:24.077167 29828 rpcz_store.cc:276] 0503 14:12:21.874295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.875416 (+ 1121us) service_pool.cc:225] Handling call
0503 14:12:23.802337 (+1926921us) ts_tablet_manager.cc:1120] Deleting tablet dda39f0323ec47799d3d44c96f3ba605
0503 14:12:23.802349 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.011530 (+209181us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.051789 (+ 40259us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.077042 (+ 25253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7488,"tablet-delete.queue_time_us":1926750}
I20240503 14:12:24.077517 29828 tablet_replica.cc:330] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:24.077690 29828 raft_consensus.cc:2229] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.077857 29828 raft_consensus.cc:2258] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.077877 28614 tablet_bootstrap.cc:654] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.078390 29828 ts_tablet_manager.cc:1875] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.090060 28614 tablet_bootstrap.cc:492] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.092190 28614 ts_tablet_manager.cc:1367] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.001s
I20240503 14:12:24.096227 29828 ts_tablet_manager.cc:1888] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:24.100653 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530f24322a454f7bb04e818ca865a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.104899 30005 consensus_queue.cc:1035] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.105764 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f74d225a13429395676d1d91253a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.096968 28614 raft_consensus.cc:348] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.122612 28614 raft_consensus.cc:374] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.122753 28614 raft_consensus.cc:729] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.123047 28614 consensus_queue.cc:260] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.123809 28614 ts_tablet_manager.cc:1398] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240503 14:12:24.124145 28614 tablet_bootstrap.cc:492] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.125598 28614 tablet_bootstrap.cc:654] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.122617 29828 log.cc:1196] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/65cdd090bf0f493f8b92341369378532
I20240503 14:12:24.127967 29828 ts_tablet_manager.cc:1909] T 65cdd090bf0f493f8b92341369378532 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:24.129037 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd28bc07e4ccd9097f6dee471c5c6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:24.129349 28489 raft_consensus.cc:3037] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.129549 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 587240644403401f8d226942b962f683 (DEFAULT_TABLE table=default.Table_715482091e944a8abe222f740b5be62b [id=37f5645a541d4c55b96766ebd210ffb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.131942 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587240644403401f8d226942b962f683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.129122 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbcae22ea124edaa14c9409f3052714" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:24.132752 28489 raft_consensus.cc:2454] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:24.137167 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 77f0081cc2cb48aaa368741109032fd1 (table default.Table_730611e6ad48480eb3f4498f3e79725b [id=ddb1f8d9ceae4da98dcb259cde8eac8c]) successfully deleted
I20240503 14:12:24.137497 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]) successfully deleted
I20240503 14:12:24.139451 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 883) took 2254 ms. Trace:
I20240503 14:12:24.139639 29828 rpcz_store.cc:276] 0503 14:12:21.884533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.884607 (+ 74us) service_pool.cc:225] Handling call
0503 14:12:24.077465 (+2192858us) ts_tablet_manager.cc:1120] Deleting tablet 65cdd090bf0f493f8b92341369378532
0503 14:12:24.077474 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.087187 (+ 9713us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.095994 (+ 8807us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.139425 (+ 43431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2192630}
I20240503 14:12:24.140839 28600 tablet_bootstrap.cc:654] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.143492 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbcae22ea124edaa14c9409f3052714" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:24.143707 28207 raft_consensus.cc:3037] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.146072 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd28bc07e4ccd9097f6dee471c5c6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:24.146261 28216 raft_consensus.cc:3037] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.151109 28216 raft_consensus.cc:2454] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:24.151839 28207 raft_consensus.cc:2454] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:24.155949 27851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu with pid 28736
I20240503 14:12:24.140408 28325 tablet_service.cc:1467] Processing CreateTablet for tablet f4f74d225a13429395676d1d91253a79 (DEFAULT_TABLE table=default.Table_30c6122e36244089a4eba499c8881c3f [id=4b1179f88f3e44269f1b582504e17d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.157529 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1bc119df7d4694bbe926611ef73245 (DEFAULT_TABLE table=default.Table_2841ebe2e868418690799c3ccd7901df [id=b292d157f70c41d7a3220b88a7cd83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.156440 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 1132014802ed4c3682769ba86312eed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.156989 28334 tablet_service.cc:1467] Processing CreateTablet for tablet d21bdd6490004300bca7a5a7fb24cd07 (DEFAULT_TABLE table=default.Table_e141d550a34a4f838e7db3335d0ea5ce [id=8b0b765740a248efa85b8ebc34f33652]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.157959 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1bc119df7d4694bbe926611ef73245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.163877 28614 tablet_bootstrap.cc:492] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.164371 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21bdd6490004300bca7a5a7fb24cd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.164057 28614 ts_tablet_manager.cc:1367] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.003s
I20240503 14:12:24.167349 28614 raft_consensus.cc:348] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.167560 28614 raft_consensus.cc:374] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.167603 28614 raft_consensus.cc:729] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.167810 28614 consensus_queue.cc:260] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.168479 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f74d225a13429395676d1d91253a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.169574 28614 ts_tablet_manager.cc:1398] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:24.169895 28614 tablet_bootstrap.cc:492] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.171998 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e14e8b5f4046029af4673a277112d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.172475 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.172999 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3896fbc6534762bfabce5050a921cc (DEFAULT_TABLE table=default.Table_60e99647c73642ceb431834009fa427e [id=5b324e27d45240bf926e6b0e764362be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.173386 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3896fbc6534762bfabce5050a921cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.173332 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f396f2a405c64cb98d09f08fedccd872"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:24.174000 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f396f2a405c64cb98d09f08fedccd872 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:24.180002 28331 tablet_service.cc:1467] Processing CreateTablet for tablet e6871d6c3610402fa13042ff86a14e11 (DEFAULT_TABLE table=default.Table_e732ed678eff478eb9c213eb0abc07bc [id=a4cd100ac6f94d1ebc017ca89bbbf852]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.180683 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6871d6c3610402fa13042ff86a14e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.182135 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 885) took 2246 ms. Trace:
I20240503 14:12:24.182309 29828 rpcz_store.cc:276] 0503 14:12:21.935383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.935682 (+ 299us) service_pool.cc:225] Handling call
0503 14:12:24.182003 (+2246321us) ts_tablet_manager.cc:1120] Deleting tablet b04fff656f624001a709f783816eec1e
0503 14:12:24.182012 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.182114 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2246124}
I20240503 14:12:24.189224 28604 tablet_bootstrap.cc:492] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.189591 28604 ts_tablet_manager.cc:1367] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.319s user 0.003s sys 0.005s
I20240503 14:12:24.190696 29828 tablet_replica.cc:330] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:24.190564 28604 raft_consensus.cc:348] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.190863 28604 raft_consensus.cc:374] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.191022 28604 raft_consensus.cc:729] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.191048 29828 raft_consensus.cc:2229] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:24.191376 29828 raft_consensus.cc:2258] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.191427 28604 consensus_queue.cc:260] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.192283 28604 ts_tablet_manager.cc:1398] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:24.192533 29828 ts_tablet_manager.cc:1875] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.191852 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet d21bdd6490004300bca7a5a7fb24cd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.194649 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_52ef5212fbac426bac3aefbc92d9de91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:24.195091 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 530f24322a454f7bb04e818ca865a369 (DEFAULT_TABLE table=default.Table_ef2d2d784a834891be22d51c1d9f7ae2 [id=a0701af20254420f952b58c1ebe298b1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:24.195390 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52ef5212fbac426bac3aefbc92d9de91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:24.195763 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530f24322a454f7bb04e818ca865a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.198464 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 24296154c38f468fbbb57dc9eaa43df2 (DEFAULT_TABLE table=default.Table_a6061a2b91e04219adf9ea0d6a4234e5 [id=5eede172e598449895c1a75445f9548e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.198989 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24296154c38f468fbbb57dc9eaa43df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.192662 28661 log.cc:1196] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/9011de10a7ff497fb35ccb3ada7315ef
I20240503 14:12:24.202152 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 799eed37c032486ab5479cfdb6d5a045 (DEFAULT_TABLE table=default.Table_4d29fd37826542f6aee3139a6338e4ae [id=a27ef895433c4455a88b25bfe79ac294]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.202827 28457 tablet_service.cc:1467] Processing CreateTablet for tablet c6ae9e2dc5604e13a13dfc998558ae6a (DEFAULT_TABLE table=default.Table_b25f8a0acb9f425aa6019f374e8bf5cc [id=18444f6fff7f430daebf6500a1bb4b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.203428 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 117935402a6e4d0d829c446a3de53782 (DEFAULT_TABLE table=default.Table_85137044068a45ce8677074d642ad58b [id=fcaf1fa27ab448d4b62dcc6ba284921f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.203779 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 77511682d71a40f5933edcb6a6523649 (DEFAULT_TABLE table=default.Table_aa95e49d837a43cf9ed5ce0f5858cc45 [id=c7e5a81bd91447a193f8b5e57ba3911a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.204273 28459 tablet_service.cc:1467] Processing CreateTablet for tablet d9f88038c4854f6190327ac19fb6bcbc (DEFAULT_TABLE table=default.Table_6650c1de6ff4404ca4f4a85acd9b2204 [id=ff540c353b014882915b8a42a29c535b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.204787 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 76d2ddc88f3045d3ade02f8460b5fb0d (DEFAULT_TABLE table=default.Table_4c7cc36935e640d0bff537b79d5b09d1 [id=6aaae2c7ccf949fd8ae1103f885908d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.205107 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca886b9403540e1aa0498a6c5e2aba3 (DEFAULT_TABLE table=default.Table_d6956bfb321b412dac6347c07258731c [id=411bde1221254e90a22b8aa57051b025]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.205883 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 19d43cc864064c4c907ec0629391da10 (DEFAULT_TABLE table=default.Table_6331f7c5c3804f44833a92e6ded96825 [id=64e016a2535a4e88a30f199c8ab99bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.206218 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 24296154c38f468fbbb57dc9eaa43df2 (DEFAULT_TABLE table=default.Table_a6061a2b91e04219adf9ea0d6a4234e5 [id=5eede172e598449895c1a75445f9548e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.208108 28661 ts_tablet_manager.cc:1909] T 9011de10a7ff497fb35ccb3ada7315ef P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.208928 28604 tablet_bootstrap.cc:492] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.209007 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f88038c4854f6190327ac19fb6bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.210605 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 19d43cc864064c4c907ec0629391da10 (DEFAULT_TABLE table=default.Table_6331f7c5c3804f44833a92e6ded96825 [id=64e016a2535a4e88a30f199c8ab99bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.210736 28604 tablet_bootstrap.cc:654] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.211099 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d43cc864064c4c907ec0629391da10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.215678 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d43cc864064c4c907ec0629391da10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.216076 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 799eed37c032486ab5479cfdb6d5a045 (DEFAULT_TABLE table=default.Table_4d29fd37826542f6aee3139a6338e4ae [id=a27ef895433c4455a88b25bfe79ac294]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.216543 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799eed37c032486ab5479cfdb6d5a045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.220479 28181 tablet_service.cc:1467] Processing CreateTablet for tablet c6ae9e2dc5604e13a13dfc998558ae6a (DEFAULT_TABLE table=default.Table_b25f8a0acb9f425aa6019f374e8bf5cc [id=18444f6fff7f430daebf6500a1bb4b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.220618 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca886b9403540e1aa0498a6c5e2aba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.220919 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ae9e2dc5604e13a13dfc998558ae6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.222021 28604 tablet_bootstrap.cc:492] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.222199 28604 ts_tablet_manager.cc:1367] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 14:12:24.223078 28604 raft_consensus.cc:348] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.223323 28604 raft_consensus.cc:374] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.223413 28604 raft_consensus.cc:729] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.223662 28604 consensus_queue.cc:260] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.224349 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ae9e2dc5604e13a13dfc998558ae6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.224437 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 77511682d71a40f5933edcb6a6523649 (DEFAULT_TABLE table=default.Table_aa95e49d837a43cf9ed5ce0f5858cc45 [id=c7e5a81bd91447a193f8b5e57ba3911a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.224855 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77511682d71a40f5933edcb6a6523649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.225431 29811 raft_consensus.cc:482] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.225561 29811 raft_consensus.cc:504] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.226738 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fad4508a29d4822b81af6947b57e7ef" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:24.227361 28150 leader_election.cc:341] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.228447 28188 tablet_service.cc:1467] Processing CreateTablet for tablet d9f88038c4854f6190327ac19fb6bcbc (DEFAULT_TABLE table=default.Table_6650c1de6ff4404ca4f4a85acd9b2204 [id=ff540c353b014882915b8a42a29c535b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.228471 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fad4508a29d4822b81af6947b57e7ef" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:24.228689 28353 raft_consensus.cc:2454] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:24.228871 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f88038c4854f6190327ac19fb6bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.229260 28150 leader_election.cc:302] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:24.229610 30010 raft_consensus.cc:2781] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:24.229681 30010 raft_consensus.cc:482] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:24.229732 30010 raft_consensus.cc:3037] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.231098 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117935402a6e4d0d829c446a3de53782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.231503 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca886b9403540e1aa0498a6c5e2aba3 (DEFAULT_TABLE table=default.Table_d6956bfb321b412dac6347c07258731c [id=411bde1221254e90a22b8aa57051b025]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.231917 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca886b9403540e1aa0498a6c5e2aba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.232256 28604 ts_tablet_manager.cc:1398] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:12:24.232431 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77511682d71a40f5933edcb6a6523649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.232622 28604 tablet_bootstrap.cc:492] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.234297 28604 tablet_bootstrap.cc:654] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.235381 29811 leader_election.cc:288] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:24.236413 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799eed37c032486ab5479cfdb6d5a045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.236485 28196 tablet_service.cc:1467] Processing CreateTablet for tablet b97da778da214bb6bc147c5ada6c7603 (DEFAULT_TABLE table=default.Table_b71e58c4b31248888eecec2d163d0181 [id=e4a9eb2d39d34430bde5d4aae9dccadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.236914 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97da778da214bb6bc147c5ada6c7603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.236881 30010 raft_consensus.cc:504] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.238071 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fad4508a29d4822b81af6947b57e7ef" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:24.238741 28150 leader_election.cc:341] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.240437 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d2ddc88f3045d3ade02f8460b5fb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.242609 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 117935402a6e4d0d829c446a3de53782 (DEFAULT_TABLE table=default.Table_85137044068a45ce8677074d642ad58b [id=fcaf1fa27ab448d4b62dcc6ba284921f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.243058 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117935402a6e4d0d829c446a3de53782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.247864 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fad4508a29d4822b81af6947b57e7ef" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:24.248132 28353 raft_consensus.cc:3037] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.248396 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24296154c38f468fbbb57dc9eaa43df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.248852 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 778) took 4653 ms. Trace:
I20240503 14:12:24.248962 28661 rpcz_store.cc:276] 0503 14:12:19.595717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.595763 (+ 46us) service_pool.cc:225] Handling call
0503 14:12:23.846673 (+4250910us) ts_tablet_manager.cc:1120] Deleting tablet 9011de10a7ff497fb35ccb3ada7315ef
0503 14:12:23.846689 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:23.936023 (+ 89334us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:23.939399 (+ 3376us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.248821 (+309422us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4250747}
I20240503 14:12:24.249295 28661 tablet_replica.cc:330] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.249497 28661 raft_consensus.cc:2229] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.249644 28661 raft_consensus.cc:2258] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.250180 28661 ts_tablet_manager.cc:1875] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.250985 28353 raft_consensus.cc:2454] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:24.251672 28150 leader_election.cc:302] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:24.252365 30005 raft_consensus.cc:2781] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.252559 30005 raft_consensus.cc:686] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:24.252920 30005 consensus_queue.cc:237] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.255445 29903 consensus_queue.cc:1035] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.257087 28661 ts_tablet_manager.cc:1888] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:24.257243 28661 log.cc:1196] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/60206e8e73e84e409ed6aba222d9ff4b
I20240503 14:12:24.257649 28661 ts_tablet_manager.cc:1909] T 60206e8e73e84e409ed6aba222d9ff4b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.258177 28185 tablet_service.cc:1467] Processing CreateTablet for tablet e750765798db487fb0b5142f01d21771 (DEFAULT_TABLE table=default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.258497 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 780) took 4655 ms. Trace:
I20240503 14:12:24.258634 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e750765798db487fb0b5142f01d21771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.258621 28661 rpcz_store.cc:276] 0503 14:12:19.602966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.603033 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:24.249245 (+4646212us) ts_tablet_manager.cc:1120] Deleting tablet 60206e8e73e84e409ed6aba222d9ff4b
0503 14:12:24.249257 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.253516 (+ 4259us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.256965 (+ 3449us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.258452 (+ 1487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4645937}
I20240503 14:12:24.258992 28661 tablet_replica.cc:330] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.259225 28661 raft_consensus.cc:2229] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:24.259450 28661 raft_consensus.cc:2258] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.260608 28614 tablet_bootstrap.cc:654] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.262398 28187 tablet_service.cc:1467] Processing CreateTablet for tablet bfc8ec1d6acb4127b7bf194226135b8c (DEFAULT_TABLE table=default.Table_85a520f4fd804e969bbb4149159976e1 [id=ed81e75ce5ae45a2ac9a547c1903c681]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.262827 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc8ec1d6acb4127b7bf194226135b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.263792 29750 raft_consensus.cc:482] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.263943 29750 raft_consensus.cc:504] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.264578 29750 leader_election.cc:288] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.260077 28661 ts_tablet_manager.cc:1875] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.263653 28604 tablet_bootstrap.cc:492] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.265514 28604 ts_tablet_manager.cc:1367] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240503 14:12:24.267586 28604 raft_consensus.cc:348] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.270264 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 76d2ddc88f3045d3ade02f8460b5fb0d (DEFAULT_TABLE table=default.Table_4c7cc36935e640d0bff537b79d5b09d1 [id=6aaae2c7ccf949fd8ae1103f885908d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.270876 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d2ddc88f3045d3ade02f8460b5fb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.273478 29750 consensus_queue.cc:1035] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.277549 28604 raft_consensus.cc:374] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.277699 28604 raft_consensus.cc:729] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.277984 28604 consensus_queue.cc:260] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.279031 28604 ts_tablet_manager.cc:1398] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:24.279949 28604 tablet_bootstrap.cc:492] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.282255 30010 leader_election.cc:288] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:24.285017 30037 tablet.cc:1722] T 02dd4c914a4846bab3a00408247ec64a P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:24.285887 28604 tablet_bootstrap.cc:654] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.258177 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 7792ad10a98946f68738ff1857edafa3 (DEFAULT_TABLE table=default.Table_865c53e9072b4f2d81c2809a817e35fd [id=0a1aec6101d54a809cde416a450e01eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.286545 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7792ad10a98946f68738ff1857edafa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.295742 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 9011de10a7ff497fb35ccb3ada7315ef (table default.Table_db04b673d60644559c17717f39eab7e2 [id=56e84e4023b242fc81a4e5c27c5ccd11]) successfully deleted
I20240503 14:12:24.296044 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 60206e8e73e84e409ed6aba222d9ff4b (table default.Table_e19b96f79a6c41c0bc266f1391941564 [id=c7144a78f2ae47599016aa64b6d0cbea]) successfully deleted
I20240503 14:12:24.297132 28600 tablet_bootstrap.cc:492] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.297317 28600 ts_tablet_manager.cc:1367] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.248s user 0.007s sys 0.007s
I20240503 14:12:24.298192 28600 raft_consensus.cc:348] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.298395 28600 raft_consensus.cc:374] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.298441 28600 raft_consensus.cc:729] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.298658 28600 consensus_queue.cc:260] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.299185 28600 ts_tablet_manager.cc:1398] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:24.299489 28600 tablet_bootstrap.cc:492] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.301563 28600 tablet_bootstrap.cc:654] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.301457 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d28d3c2ea472191ad21355248a2fc" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:24.301679 28349 raft_consensus.cc:3037] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.304978 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cd09e769be4bee95bbef9c3c550451" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:24.307119 28462 tablet_service.cc:1467] Processing CreateTablet for tablet bfc8ec1d6acb4127b7bf194226135b8c (DEFAULT_TABLE table=default.Table_85a520f4fd804e969bbb4149159976e1 [id=ed81e75ce5ae45a2ac9a547c1903c681]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.307670 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc8ec1d6acb4127b7bf194226135b8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:24.311291 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a (table default.Table_79df8c9603ae456799a73fb5badd18e9 [id=23dabedfe8df4e9fa6123f2dd9b82376], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5dd8b2adaeeb4c3abfbfe68cc005e03a consensus state has no leader
I20240503 14:12:24.316411 28457 tablet_service.cc:1467] Processing CreateTablet for tablet e750765798db487fb0b5142f01d21771 (DEFAULT_TABLE table=default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.317152 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e750765798db487fb0b5142f01d21771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.320597 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 7792ad10a98946f68738ff1857edafa3 (DEFAULT_TABLE table=default.Table_865c53e9072b4f2d81c2809a817e35fd [id=0a1aec6101d54a809cde416a450e01eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.321326 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7792ad10a98946f68738ff1857edafa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.294495 28661 ts_tablet_manager.cc:1888] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:24.324635 28661 log.cc:1196] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/77f0081cc2cb48aaa368741109032fd1
I20240503 14:12:24.325157 28661 ts_tablet_manager.cc:1909] T 77f0081cc2cb48aaa368741109032fd1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.326133 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 781) took 4722 ms. Trace:
I20240503 14:12:24.326321 28661 rpcz_store.cc:276] 0503 14:12:19.603903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.603953 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:24.258923 (+4654970us) ts_tablet_manager.cc:1120] Deleting tablet 77f0081cc2cb48aaa368741109032fd1
0503 14:12:24.258933 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.282076 (+ 23143us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.294338 (+ 12262us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.326099 (+ 31761us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4654710}
I20240503 14:12:24.326845 28661 tablet_replica.cc:330] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.327106 28661 raft_consensus.cc:2229] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.327318 28661 raft_consensus.cc:2258] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.327971 28661 ts_tablet_manager.cc:1875] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.332578 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet abb9279e91124fe6bfbdc8856eccfed8
I20240503 14:12:24.333125 28032 catalog_manager.cc:6554] ReplaceTablet: tablet abb9279e91124fe6bfbdc8856eccfed8 deleted and replaced by tablet 79a96d73b2d142f0b6ada0c1c1cc654e
I20240503 14:12:24.333276 28600 tablet_bootstrap.cc:492] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.333426 28600 ts_tablet_manager.cc:1367] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.002s
I20240503 14:12:24.333397 29828 ts_tablet_manager.cc:1888] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:24.333531 29828 log.cc:1196] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/41cbd1326869434594e126421ad132d8
W20240503 14:12:24.333933 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99aa7e59c45c45c7bc1f726c97426b89 consensus state has no leader
I20240503 14:12:24.334360 28600 raft_consensus.cc:348] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.334571 28600 raft_consensus.cc:374] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.334625 28600 raft_consensus.cc:729] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.334908 28600 consensus_queue.cc:260] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.335486 28600 ts_tablet_manager.cc:1398] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:24.335813 28600 tablet_bootstrap.cc:492] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.339802 28604 tablet_bootstrap.cc:492] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.340245 28604 ts_tablet_manager.cc:1367] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.001s
I20240503 14:12:24.342716 29828 ts_tablet_manager.cc:1909] T 41cbd1326869434594e126421ad132d8 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:24.343601 28614 tablet_bootstrap.cc:492] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.343783 28614 ts_tablet_manager.cc:1367] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.174s user 0.004s sys 0.000s
I20240503 14:12:24.342764 29750 raft_consensus.cc:482] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.345153 29750 raft_consensus.cc:504] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.345434 28604 raft_consensus.cc:348] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.345669 28604 raft_consensus.cc:374] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.345721 28604 raft_consensus.cc:729] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.345846 29750 leader_election.cc:288] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.345185 28614 raft_consensus.cc:348] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.345952 28604 consensus_queue.cc:260] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.346087 28614 raft_consensus.cc:374] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.346145 28614 raft_consensus.cc:729] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.328203 28472 tablet_service.cc:1467] Processing CreateTablet for tablet b97da778da214bb6bc147c5ada6c7603 (DEFAULT_TABLE table=default.Table_b71e58c4b31248888eecec2d163d0181 [id=e4a9eb2d39d34430bde5d4aae9dccadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.346774 28604 ts_tablet_manager.cc:1398] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:24.347100 28604 tablet_bootstrap.cc:492] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.347532 29864 raft_consensus.cc:482] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.347690 29864 raft_consensus.cc:504] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.348402 29864 leader_election.cc:288] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.347558 30015 raft_consensus.cc:482] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.348635 30015 raft_consensus.cc:504] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.349540 30015 leader_election.cc:288] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.350426 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cd09e769be4bee95bbef9c3c550451" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:24.350674 28352 raft_consensus.cc:2379] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:24.354983 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 02dd4c914a4846bab3a00408247ec64a (table default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc]). Expected version 2 got 0
I20240503 14:12:24.355593 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet e94673f00cb3432185009bda2ce398fa (table default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d [id=2849f6ba9ef64d26a5dc65f9ac9048da]). Expected version 1 got 0
I20240503 14:12:24.355995 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:24.358047 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2c1e6032d7f8400d84b575d7c1c08e32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:24.358871 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c1e6032d7f8400d84b575d7c1c08e32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:24.361663 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c2b8fa4eaa2946d8b7f1bd78f114e8fe" } new_table_name: "default.Table_881b9fb0c89441bcb58040654dc3db10" modify_external_catalogs: true
I20240503 14:12:24.363986 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_59522dec1afc43cbb814f4ecb19d9a16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:24.364713 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59522dec1afc43cbb814f4ecb19d9a16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:24.366736 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_29aaa3a8d8ed47dab6dfc37b3989f427" } new_table_name: "default.Table_bb7c604acc7e4f70bfadef3d616c95c1" modify_external_catalogs: true
I20240503 14:12:24.370909 28489 raft_consensus.cc:3037] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.373157 28600 tablet_bootstrap.cc:654] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.378192 28614 consensus_queue.cc:260] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.375686 29811 consensus_queue.cc:1035] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.378949 28614 ts_tablet_manager.cc:1398] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240503 14:12:24.379300 28614 tablet_bootstrap.cc:492] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.347329 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97da778da214bb6bc147c5ada6c7603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.380137 28604 tablet_bootstrap.cc:654] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.389402 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet abb9279e91124fe6bfbdc8856eccfed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 79a96d73b2d142f0b6ada0c1c1cc654e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:24.393707 28661 ts_tablet_manager.cc:1888] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:24.395079 28600 tablet_bootstrap.cc:492] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.395381 28600 ts_tablet_manager.cc:1367] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20240503 14:12:24.396379 28600 raft_consensus.cc:348] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.396754 28600 raft_consensus.cc:374] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.396864 28600 raft_consensus.cc:729] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.397231 28600 consensus_queue.cc:260] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.398326 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe3affd0d144b1fb0078e7690b9b36d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:24.398712 28209 raft_consensus.cc:2454] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:24.398998 28600 ts_tablet_manager.cc:1398] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:24.399510 28600 tablet_bootstrap.cc:492] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.399935 30015 consensus_queue.cc:1035] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.400750 28290 leader_election.cc:302] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:24.401111 30015 raft_consensus.cc:2781] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.401247 30015 raft_consensus.cc:686] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:24.401500 30015 consensus_queue.cc:237] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.390067 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 65cdd090bf0f493f8b92341369378532 (table default.Table_1ce31ad9481d4ef7ac7dc40df1c5fc6c [id=d16dae55faab45f4bb6337e042735ad0]) successfully deleted
W20240503 14:12:24.408824 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b04fff656f624001a709f783816eec1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b04fff656f624001a709f783816eec1e already in progress: creating tablet
I20240503 14:12:24.410295 28614 tablet_bootstrap.cc:654] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.412123 28349 raft_consensus.cc:2454] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:24.412976 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet abb9279e91124fe6bfbdc8856eccfed8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abb9279e91124fe6bfbdc8856eccfed8 already in progress: creating tablet
I20240503 14:12:24.413578 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 77f0081cc2cb48aaa368741109032fd1 (table default.Table_730611e6ad48480eb3f4498f3e79725b [id=ddb1f8d9ceae4da98dcb259cde8eac8c]) successfully deleted
I20240503 14:12:24.405859 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd8fb8ea1a4fc3bfa4f55ea59ae12a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:24.417747 28600 tablet_bootstrap.cc:654] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.397945 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd8fb8ea1a4fc3bfa4f55ea59ae12a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:24.398454 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe3affd0d144b1fb0078e7690b9b36d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:24.418221 28491 raft_consensus.cc:2454] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:24.408632 28604 tablet_bootstrap.cc:492] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.418866 28604 ts_tablet_manager.cc:1367] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.001s
I20240503 14:12:24.412853 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 884) took 2477 ms. Trace:
I20240503 14:12:24.419716 29828 rpcz_store.cc:276] 0503 14:12:21.935288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.935944 (+ 656us) service_pool.cc:225] Handling call
0503 14:12:24.190633 (+2254689us) ts_tablet_manager.cc:1120] Deleting tablet 41cbd1326869434594e126421ad132d8
0503 14:12:24.190645 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.321574 (+130929us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.333261 (+ 11687us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.412827 (+ 79566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254535}
I20240503 14:12:24.419790 28604 raft_consensus.cc:348] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.419978 28604 raft_consensus.cc:374] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.420027 28604 raft_consensus.cc:729] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.420255 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 904) took 1292 ms. Trace:
I20240503 14:12:24.420346 29828 rpcz_store.cc:276] 0503 14:12:23.127639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.127699 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:24.420063 (+1292364us) ts_tablet_manager.cc:1120] Deleting tablet 00a56d79c36f4ffa9bae02e2369d8ded
0503 14:12:24.420077 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.420229 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292174}
I20240503 14:12:24.420682 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 905) took 1285 ms. Trace:
I20240503 14:12:24.420644 28604 consensus_queue.cc:260] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.420764 29828 rpcz_store.cc:276] 0503 14:12:23.135728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.135801 (+ 73us) service_pool.cc:225] Handling call
0503 14:12:24.420570 (+1284769us) ts_tablet_manager.cc:1120] Deleting tablet 380f267a984541c7878eeaa4eeee8f34
0503 14:12:24.420580 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.420664 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1284521}
I20240503 14:12:24.421113 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 906) took 1285 ms. Trace:
I20240503 14:12:24.421228 28604 ts_tablet_manager.cc:1398] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:24.421180 29828 rpcz_store.cc:276] 0503 14:12:23.135862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.135911 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:24.420988 (+1285077us) ts_tablet_manager.cc:1120] Deleting tablet 0c300b1f104949b3ae74f5b38d378a11
0503 14:12:24.421018 (+ 30us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.421098 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10340,"tablet-delete.queue_time_us":1284868}
I20240503 14:12:24.421478 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 907) took 1285 ms. Trace:
I20240503 14:12:24.421515 28604 tablet_bootstrap.cc:492] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.421538 29828 rpcz_store.cc:276] 0503 14:12:23.136361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.136410 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:24.421384 (+1284974us) ts_tablet_manager.cc:1120] Deleting tablet 86804987bdf340dc83ddd58ba2bc8a96
0503 14:12:24.421392 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.421463 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9932,"tablet-delete.queue_time_us":1284815}
I20240503 14:12:24.421790 29828 tablet_replica.cc:330] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:24.421974 29828 raft_consensus.cc:2229] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.422142 29828 raft_consensus.cc:2258] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.422147 28661 log.cc:1196] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/2331aec871824a238f3f64de217d7092
I20240503 14:12:24.422523 28661 ts_tablet_manager.cc:1909] T 2331aec871824a238f3f64de217d7092 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.422742 29828 ts_tablet_manager.cc:1875] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.429618 28604 tablet_bootstrap.cc:654] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:24.430366 28427 consensus_peers.cc:485] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dda39f0323ec47799d3d44c96f3ba605. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:24.395018 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbcae22ea124edaa14c9409f3052714" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:24.431088 28207 raft_consensus.cc:2379] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:24.432950 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 782) took 4681 ms. Trace:
I20240503 14:12:24.433128 28661 rpcz_store.cc:276] 0503 14:12:19.751387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:19.751847 (+ 460us) service_pool.cc:225] Handling call
0503 14:12:24.326788 (+4574941us) ts_tablet_manager.cc:1120] Deleting tablet 2331aec871824a238f3f64de217d7092
0503 14:12:24.326799 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.374813 (+ 48014us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.393491 (+ 18678us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.432924 (+ 39433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4574638}
I20240503 14:12:24.433691 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 807) took 3991 ms. Trace:
I20240503 14:12:24.433863 28661 rpcz_store.cc:276] 0503 14:12:20.441880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.442049 (+ 169us) service_pool.cc:225] Handling call
0503 14:12:24.433542 (+3991493us) ts_tablet_manager.cc:1120] Deleting tablet be238ee1d82548c9a1940e2a72e395c7
0503 14:12:24.433553 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.433669 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3991227}
I20240503 14:12:24.434345 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 808) took 3991 ms. Trace:
I20240503 14:12:24.434674 28661 rpcz_store.cc:276] 0503 14:12:20.442973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.443285 (+ 312us) service_pool.cc:225] Handling call
0503 14:12:24.434218 (+3990933us) ts_tablet_manager.cc:1120] Deleting tablet 332ecc478e49424d9e7d7decbbc2c29c
0503 14:12:24.434227 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.434326 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3990711}
I20240503 14:12:24.435117 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 810) took 3938 ms. Trace:
I20240503 14:12:24.435251 28290 leader_election.cc:302] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:24.435258 28661 rpcz_store.cc:276] 0503 14:12:20.496864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.497123 (+ 259us) service_pool.cc:225] Handling call
0503 14:12:24.435004 (+3937881us) ts_tablet_manager.cc:1120] Deleting tablet 6615052ddbde487e840b6fd77a140e58
0503 14:12:24.435014 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.435101 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3937573}
I20240503 14:12:24.435604 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 811) took 3938 ms. Trace:
I20240503 14:12:24.435708 29864 raft_consensus.cc:2781] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.435842 29864 raft_consensus.cc:686] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:24.435757 28661 rpcz_store.cc:276] 0503 14:12:20.496972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.497488 (+ 516us) service_pool.cc:225] Handling call
0503 14:12:24.435468 (+3937980us) ts_tablet_manager.cc:1120] Deleting tablet f5bea5960cbb44d58d1dbabc888903c2
0503 14:12:24.435477 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.435589 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3937856}
I20240503 14:12:24.436120 29864 consensus_queue.cc:237] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.436450 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 812) took 3938 ms. Trace:
I20240503 14:12:24.436571 28661 rpcz_store.cc:276] 0503 14:12:20.497832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.498313 (+ 481us) service_pool.cc:225] Handling call
0503 14:12:24.436271 (+3937958us) ts_tablet_manager.cc:1120] Deleting tablet b86c2df0470a4945b3e8207f2d965de2
0503 14:12:24.436279 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.436432 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3937743}
I20240503 14:12:24.436985 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 809) took 3940 ms. Trace:
I20240503 14:12:24.437129 28661 rpcz_store.cc:276] 0503 14:12:20.496737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.500310 (+ 3573us) service_pool.cc:225] Handling call
0503 14:12:24.436877 (+3936567us) ts_tablet_manager.cc:1120] Deleting tablet a31555fc800142bd849dbd572a194002
0503 14:12:24.436886 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.436967 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3936336}
I20240503 14:12:24.437541 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 813) took 3910 ms. Trace:
I20240503 14:12:24.437678 28661 rpcz_store.cc:276] 0503 14:12:20.526850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.526915 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:24.437433 (+3910518us) ts_tablet_manager.cc:1120] Deleting tablet e708be06de744743aac4e192bdd0f449
0503 14:12:24.437441 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.437524 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3910277}
I20240503 14:12:24.438071 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 814) took 3862 ms. Trace:
I20240503 14:12:24.438207 28661 rpcz_store.cc:276] 0503 14:12:20.575898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.575962 (+ 64us) service_pool.cc:225] Handling call
0503 14:12:24.437973 (+3862011us) ts_tablet_manager.cc:1120] Deleting tablet dcd405fd84d44076b1792b7fcbe00e84
0503 14:12:24.437981 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.438055 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3861730}
I20240503 14:12:24.438592 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 825) took 3719 ms. Trace:
I20240503 14:12:24.438719 28661 rpcz_store.cc:276] 0503 14:12:20.719308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.719527 (+ 219us) service_pool.cc:225] Handling call
0503 14:12:24.438494 (+3718967us) ts_tablet_manager.cc:1120] Deleting tablet cf221f17ddfd461f83f7f916d7833011
0503 14:12:24.438503 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.438576 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3700335}
I20240503 14:12:24.439311 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 839) took 3669 ms. Trace:
I20240503 14:12:24.439978 28661 rpcz_store.cc:276] 0503 14:12:20.769797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.769997 (+ 200us) service_pool.cc:225] Handling call
0503 14:12:24.439209 (+3669212us) ts_tablet_manager.cc:1120] Deleting tablet a3d1314841574fc8bcf3ccb16755e421
0503 14:12:24.439219 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.439296 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3668260}
I20240503 14:12:24.440824 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 837) took 3701 ms. Trace:
I20240503 14:12:24.440945 28614 tablet_bootstrap.cc:492] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.441166 28614 ts_tablet_manager.cc:1367] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.001s
I20240503 14:12:24.442167 28614 raft_consensus.cc:348] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.442467 28614 raft_consensus.cc:374] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.442518 28614 raft_consensus.cc:729] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.442732 28614 consensus_queue.cc:260] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.443325 29828 ts_tablet_manager.cc:1888] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:24.443535 29828 log.cc:1196] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ed4dc732bcb44222b4b15e5d77defe33
I20240503 14:12:24.443933 29828 ts_tablet_manager.cc:1909] T ed4dc732bcb44222b4b15e5d77defe33 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:24.444380 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 908) took 1304 ms. Trace:
I20240503 14:12:24.444536 29828 rpcz_store.cc:276] 0503 14:12:23.140385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.140462 (+ 77us) service_pool.cc:225] Handling call
0503 14:12:24.421744 (+1281282us) ts_tablet_manager.cc:1120] Deleting tablet ed4dc732bcb44222b4b15e5d77defe33
0503 14:12:24.421753 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.434841 (+ 13088us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.443200 (+ 8359us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.444356 (+ 1156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5858,"tablet-delete.queue_time_us":1281071}
I20240503 14:12:24.451851 28661 rpcz_store.cc:276] 0503 14:12:20.739727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.739770 (+ 43us) service_pool.cc:225] Handling call
0503 14:12:24.440697 (+3700927us) ts_tablet_manager.cc:1120] Deleting tablet a53279669bad40ad9243f4554a4e3c70
0503 14:12:24.440706 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.440808 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3665011}
I20240503 14:12:24.452519 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 836) took 3712 ms. Trace:
I20240503 14:12:24.451656 28600 tablet_bootstrap.cc:492] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.452944 28600 ts_tablet_manager.cc:1367] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.003s
I20240503 14:12:24.453919 28600 raft_consensus.cc:348] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.454231 28661 rpcz_store.cc:276] 0503 14:12:20.739636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.739677 (+ 41us) service_pool.cc:225] Handling call
0503 14:12:24.452355 (+3712678us) ts_tablet_manager.cc:1120] Deleting tablet 4e11137b75794e2c9dd61ded326c868f
0503 14:12:24.452366 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.452497 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3668480}
I20240503 14:12:24.454811 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 838) took 3685 ms. Trace:
I20240503 14:12:24.449321 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet abb9279e91124fe6bfbdc8856eccfed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 79a96d73b2d142f0b6ada0c1c1cc654e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:24.460179 28661 rpcz_store.cc:276] 0503 14:12:20.769593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.769667 (+ 74us) service_pool.cc:225] Handling call
0503 14:12:24.454627 (+3684960us) ts_tablet_manager.cc:1120] Deleting tablet 6f0d676fa1e74ab89922f7ba607fd2d8
0503 14:12:24.454639 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.454790 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3668562}
I20240503 14:12:24.460958 28600 raft_consensus.cc:374] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.443352 28614 ts_tablet_manager.cc:1398] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:24.460425 29903 raft_consensus.cc:482] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.461433 29955 raft_consensus.cc:482] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.461536 28614 tablet_bootstrap.cc:492] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.461429 29903 raft_consensus.cc:504] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.461584 29955 raft_consensus.cc:504] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.462198 29955 leader_election.cc:288] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.463271 28614 tablet_bootstrap.cc:654] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.463436 29903 leader_election.cc:288] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:24.464471 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:24.465185 28150 leader_election.cc:341] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.465301 30059 tablet.cc:1722] T 240abf8c2b2343408fc98a8dac1fdf09 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:24.463275 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:24.466303 28150 leader_election.cc:341] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.466423 28150 leader_election.cc:302] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:24.467101 29903 raft_consensus.cc:2726] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:24.467999 28661 tablet_replica.cc:330] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.461134 28600 raft_consensus.cc:729] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.468358 28661 raft_consensus.cc:2229] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:24.469187 28600 consensus_queue.cc:260] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.469477 28290 leader_election.cc:302] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:24.469692 28661 pending_rounds.cc:62] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:24.469852 29864 raft_consensus.cc:2781] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:24.470000 29864 raft_consensus.cc:482] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:24.470139 29864 raft_consensus.cc:3037] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:24.475530 28290 leader_election.cc:341] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.473618 29994 consensus_queue.cc:1035] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.478444 29949 consensus_queue.cc:1035] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.478629 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet 49445f0f1dd1451c9ba066e61273f2f1 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:24.479171 28661 pending_rounds.cc:68] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597739070947328 op_type: NO_OP noop_request { }
I20240503 14:12:24.479415 28661 raft_consensus.cc:2866] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240503 14:12:24.479642 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59b7628bcae140939184428ed15dda93 consensus state has no leader
I20240503 14:12:24.479766 28600 ts_tablet_manager.cc:1398] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240503 14:12:24.481045 28661 raft_consensus.cc:2258] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.481680 28661 ts_tablet_manager.cc:1875] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.480662 28427 leader_election.cc:302] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:24.480940 28604 tablet_bootstrap.cc:492] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.483067 28604 ts_tablet_manager.cc:1367] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.005s
I20240503 14:12:24.484071 28604 raft_consensus.cc:348] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.484470 28604 raft_consensus.cc:374] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.484576 28604 raft_consensus.cc:729] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.484861 28604 consensus_queue.cc:260] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.485666 28604 ts_tablet_manager.cc:1398] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:24.486212 28600 tablet_bootstrap.cc:492] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.488109 28604 tablet_bootstrap.cc:492] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.489914 28604 tablet_bootstrap.cc:654] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.492185 29750 raft_consensus.cc:3037] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.493773 28604 tablet_bootstrap.cc:492] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.493916 28604 ts_tablet_manager.cc:1367] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:24.494537 28600 tablet_bootstrap.cc:654] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.494765 29750 raft_consensus.cc:2726] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:24.494714 28604 raft_consensus.cc:348] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.494899 28604 raft_consensus.cc:374] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.494944 28604 raft_consensus.cc:729] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.495151 28604 consensus_queue.cc:260] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.498080 28604 ts_tablet_manager.cc:1398] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:24.498875 28604 tablet_bootstrap.cc:492] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.499135 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d62c11b873954239b3c4c78e716c2144"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:24.499440 28661 ts_tablet_manager.cc:1888] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:24.499550 28661 log.cc:1196] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/95c97a43bd714da68c1a77408f3d1cd1
W20240503 14:12:24.499866 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d62c11b873954239b3c4c78e716c2144 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:24.499917 28661 ts_tablet_manager.cc:1909] T 95c97a43bd714da68c1a77408f3d1cd1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.500553 28604 tablet_bootstrap.cc:654] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.501646 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f6b29fc1fc8b4818a92c622169ec4bd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:24.502275 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6b29fc1fc8b4818a92c622169ec4bd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:24.504046 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d6e534b5b7fc4dfcb5c7170d086552f8" } modify_external_catalogs: true
I20240503 14:12:24.504294 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d6e534b5b7fc4dfcb5c7170d086552f8" } modify_external_catalogs: true
I20240503 14:12:24.506294 29994 raft_consensus.cc:482] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.506454 29994 raft_consensus.cc:504] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.507134 29994 leader_election.cc:288] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.510923 29903 raft_consensus.cc:482] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.511195 29903 raft_consensus.cc:504] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.512490 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bba6a3d6e84552b027f23a29cbb38a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:24.512730 28491 raft_consensus.cc:2454] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:24.513274 28150 leader_election.cc:302] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:24.513635 29811 raft_consensus.cc:2781] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:24.513757 29811 raft_consensus.cc:482] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:24.513828 29811 raft_consensus.cc:3037] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.514050 29903 leader_election.cc:288] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:24.514447 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:24.514741 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet e94673f00cb3432185009bda2ce398fa (table default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d [id=2849f6ba9ef64d26a5dc65f9ac9048da]). Expected version 1 got 0
I20240503 14:12:24.515190 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 27e64d1f0cb341c7ba46def638ffe2ae (DEFAULT_TABLE table=default.Table_c2cb53287e4e4b3781e13698a10d0f43 [id=19f530d8163344159cfd015cf3d6f94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.515650 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e64d1f0cb341c7ba46def638ffe2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.516959 29811 raft_consensus.cc:504] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.518323 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bba6a3d6e84552b027f23a29cbb38a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:24.518543 28491 raft_consensus.cc:3037] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.518824 28025 catalog_manager.cc:5484] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:24.519276 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet e94673f00cb3432185009bda2ce398fa (table default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d [id=2849f6ba9ef64d26a5dc65f9ac9048da]). Expected version 1 got 0
I20240503 14:12:24.519891 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c] alter complete (version 3)
I20240503 14:12:24.520517 28028 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_5dcdff74f1f7407084a45dc9fe588103 [id=47ade8624b00424d8fc04dbca5174ff4] alter complete (version 3)
I20240503 14:12:24.521366 28491 raft_consensus.cc:2454] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:24.522213 28150 leader_election.cc:302] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:24.522274 28024 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_4d92feade7db42d483f2f56f9788f20f [id=b6e0d36bceef43e7810af395e207d8bc] alter complete (version 2)
I20240503 14:12:24.522784 29811 leader_election.cc:288] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:24.523033 29811 raft_consensus.cc:2781] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.523206 29811 raft_consensus.cc:686] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:24.523744 29811 consensus_queue.cc:237] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.526669 28196 tablet_service.cc:1467] Processing CreateTablet for tablet ed0f4a763f0c40479226f0d11323e141 (DEFAULT_TABLE table=default.Table_023451ab297b451da33176a3d13fe295 [id=396cea538c1141618c7cda8c79e3c1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.527626 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0f4a763f0c40479226f0d11323e141. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:24.528026 28028 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 104) took 1831 ms (client timeout 1000 ms). Trace:
W20240503 14:12:24.528177 28028 rpcz_store.cc:269] 0503 14:12:22.696949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.730893 (+ 33944us) service_pool.cc:225] Handling call
0503 14:12:22.742483 (+ 11590us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:24.527946 (+1785463us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.373820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.384003 (+ 10183us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.384011 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:23.384014 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.384147 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.384160 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.384165 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:23.384238 (+ 73us) tablet.cc:699] Row locks acquired
0503 14:12:23.384241 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:23.384287 (+ 46us) write_op.cc:270] Start()
0503 14:12:23.384323 (+ 36us) write_op.cc:276] Timestamp: P: 1714745543384239 usec, L: 0
0503 14:12:23.426525 (+ 42202us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.426800 (+ 275us) log.cc:842] Serialized 356 byte log entry
0503 14:12:23.519690 (+ 92890us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.520388 (+ 698us) write_op.cc:301] APPLY: starting
0503 14:12:23.520407 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.520448 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.520481 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.520540 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.520574 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:23.520581 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:23.520661 (+ 80us) log.cc:842] Serialized 23 byte log entry
0503 14:12:23.520735 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.520755 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:23.520762 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.520816 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.520819 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"spinlock_wait_cycles":24747776,"child_traces":[["op",{"apply.queue_time_us":628,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":485,"num_ops":1,"prepare.queue_time_us":328109,"prepare.run_cpu_time_us":885,"prepare.run_wall_time_us":86974,"raft.queue_time_us":36386,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":92353}]]}
I20240503 14:12:24.532261 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bba6a3d6e84552b027f23a29cbb38a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:24.536430 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 312857b304e04b18b47a4ed6980a5cbf (DEFAULT_TABLE table=default.Table_5fb312c322b34390a3823a7d55e21ebe [id=443266ed484a40c6b15c0098baaeb1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.536916 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312857b304e04b18b47a4ed6980a5cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.536461 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 841) took 3626 ms. Trace:
I20240503 14:12:24.544173 28661 rpcz_store.cc:276] 0503 14:12:20.910062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.910557 (+ 495us) service_pool.cc:225] Handling call
0503 14:12:24.467944 (+3557387us) ts_tablet_manager.cc:1120] Deleting tablet 95c97a43bd714da68c1a77408f3d1cd1
0503 14:12:24.467955 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.496369 (+ 28414us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.499326 (+ 2957us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.536429 (+ 37103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3557150}
I20240503 14:12:24.544653 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e20c31514a2a4f089d76d0d26ece4265" } new_table_name: "default.Table_7d54afe4afa34f16bd9e0d4e0c61ab72" modify_external_catalogs: true
I20240503 14:12:24.544813 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 842) took 3630 ms. Trace:
I20240503 14:12:24.544899 28661 rpcz_store.cc:276] 0503 14:12:20.914277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.914365 (+ 88us) service_pool.cc:225] Handling call
0503 14:12:24.544672 (+3630307us) ts_tablet_manager.cc:1120] Deleting tablet 1603e2360041427f8635782e812b6431
0503 14:12:24.544684 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.544790 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3630065}
I20240503 14:12:24.545187 28661 tablet_replica.cc:330] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.545390 28661 raft_consensus.cc:2229] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.545562 28661 raft_consensus.cc:2258] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.546176 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4ebf150e25b946eab62caf24e5a3c292" } modify_external_catalogs: true
I20240503 14:12:24.546386 28661 ts_tablet_manager.cc:1875] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.546397 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4ebf150e25b946eab62caf24e5a3c292" } modify_external_catalogs: true
I20240503 14:12:24.550032 30056 tablet.cc:1722] T 240abf8c2b2343408fc98a8dac1fdf09 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:24.551198 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 464f9dd19dfb4447b5faebc974671601 (DEFAULT_TABLE table=default.Table_d0f835c02d5f4ffea64369bb8db62de9 [id=fb7f675798544e498af840f47b553dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.551828 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f9dd19dfb4447b5faebc974671601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.561405 28180 tablet_service.cc:1467] Processing CreateTablet for tablet e8ee387133b6459e9e4730793c92fef8 (DEFAULT_TABLE table=default.Table_eeead1b7b6d54b15845ec251e33f6ca8 [id=ac8ee6ed937244b8b732204c0736a2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.561909 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ee387133b6459e9e4730793c92fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.532560 28349 raft_consensus.cc:2454] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:24.563235 29864 raft_consensus.cc:504] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.541904 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bba6a3d6e84552b027f23a29cbb38a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:24.563860 28355 raft_consensus.cc:3037] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.564049 29864 leader_election.cc:288] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.567142 28355 raft_consensus.cc:2454] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:24.571918 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920be6d87c4440d0bf7453c86cfe0b5c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:24.573136 28661 ts_tablet_manager.cc:1888] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:24.573292 28661 log.cc:1196] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7abcb8658fc34c939aa74cf0bd981ae4
I20240503 14:12:24.573626 28661 ts_tablet_manager.cc:1909] T 7abcb8658fc34c939aa74cf0bd981ae4 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.576162 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 (DEFAULT_TABLE table=default.Table_8220e8ba6ccb4f89ba7481c40f8af3ed [id=dd61cf2d027144bf91d38d396333958c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.576656 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5087cf4a795f4ee08ce4dd19d095d0e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:24.580200 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 101850us
W20240503 14:12:24.580538 28290 leader_election.cc:341] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.580621 28614 tablet_bootstrap.cc:492] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.580739 28600 tablet_bootstrap.cc:492] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.580785 28614 ts_tablet_manager.cc:1367] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.119s user 0.000s sys 0.003s
I20240503 14:12:24.580896 28600 ts_tablet_manager.cc:1367] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.095s user 0.010s sys 0.005s
W20240503 14:12:24.581221 28290 leader_election.cc:341] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.581353 28290 leader_election.cc:302] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:24.581578 29949 raft_consensus.cc:2726] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:24.582202 28600 raft_consensus.cc:348] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.582389 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 24296154c38f468fbbb57dc9eaa43df2 (DEFAULT_TABLE table=default.Table_a6061a2b91e04219adf9ea0d6a4234e5 [id=5eede172e598449895c1a75445f9548e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.582484 28600 raft_consensus.cc:374] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.582540 28600 raft_consensus.cc:729] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.582832 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24296154c38f468fbbb57dc9eaa43df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.582770 28600 consensus_queue.cc:260] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.583359 28600 ts_tablet_manager.cc:1398] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:24.581573 28614 raft_consensus.cc:348] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.584362 28614 raft_consensus.cc:374] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.584424 28614 raft_consensus.cc:729] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.584635 28614 consensus_queue.cc:260] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.585206 28614 ts_tablet_manager.cc:1398] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:12:24.585531 28614 tablet_bootstrap.cc:492] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.585628 28600 tablet_bootstrap.cc:492] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.589083 28600 tablet_bootstrap.cc:654] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.598213 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 76d2ddc88f3045d3ade02f8460b5fb0d (DEFAULT_TABLE table=default.Table_4c7cc36935e640d0bff537b79d5b09d1 [id=6aaae2c7ccf949fd8ae1103f885908d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.598445 28336 tablet_service.cc:1467] Processing CreateTablet for tablet 77511682d71a40f5933edcb6a6523649 (DEFAULT_TABLE table=default.Table_aa95e49d837a43cf9ed5ce0f5858cc45 [id=c7e5a81bd91447a193f8b5e57ba3911a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.598752 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d2ddc88f3045d3ade02f8460b5fb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.598920 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77511682d71a40f5933edcb6a6523649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.600394 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 19d43cc864064c4c907ec0629391da10 (DEFAULT_TABLE table=default.Table_6331f7c5c3804f44833a92e6ded96825 [id=64e016a2535a4e88a30f199c8ab99bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.600776 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d43cc864064c4c907ec0629391da10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.604444 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 799eed37c032486ab5479cfdb6d5a045 (DEFAULT_TABLE table=default.Table_4d29fd37826542f6aee3139a6338e4ae [id=a27ef895433c4455a88b25bfe79ac294]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.604858 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799eed37c032486ab5479cfdb6d5a045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.604794 28324 tablet_service.cc:1467] Processing CreateTablet for tablet d9f88038c4854f6190327ac19fb6bcbc (DEFAULT_TABLE table=default.Table_6650c1de6ff4404ca4f4a85acd9b2204 [id=ff540c353b014882915b8a42a29c535b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.605094 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f88038c4854f6190327ac19fb6bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.605355 28334 tablet_service.cc:1467] Processing CreateTablet for tablet c6ae9e2dc5604e13a13dfc998558ae6a (DEFAULT_TABLE table=default.Table_b25f8a0acb9f425aa6019f374e8bf5cc [id=18444f6fff7f430daebf6500a1bb4b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.605940 28323 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca886b9403540e1aa0498a6c5e2aba3 (DEFAULT_TABLE table=default.Table_d6956bfb321b412dac6347c07258731c [id=411bde1221254e90a22b8aa57051b025]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.608690 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca886b9403540e1aa0498a6c5e2aba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.609294 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ae9e2dc5604e13a13dfc998558ae6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.612313 29811 raft_consensus.cc:482] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.612512 29811 raft_consensus.cc:504] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:24.613329 29811 leader_election.cc:288] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:24.614097 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b176a071ba4f3fbff2a64d6e46027a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:24.614297 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b176a071ba4f3fbff2a64d6e46027a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:24.614914 28150 leader_election.cc:341] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:24.615402 28150 leader_election.cc:341] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.615504 28150 leader_election.cc:302] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:24.617686 28600 tablet_bootstrap.cc:492] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.617864 28600 ts_tablet_manager.cc:1367] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s
I20240503 14:12:24.618666 28600 raft_consensus.cc:348] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.618888 28600 raft_consensus.cc:374] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.618937 28600 raft_consensus.cc:729] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.619199 28600 consensus_queue.cc:260] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.619505 29903 raft_consensus.cc:2726] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:24.621677 28614 tablet_bootstrap.cc:654] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.622543 28600 ts_tablet_manager.cc:1398] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:24.622929 28600 tablet_bootstrap.cc:492] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.632018 28600 tablet_bootstrap.cc:654] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.640748 29811 consensus_queue.cc:1035] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.642868 28600 tablet_bootstrap.cc:492] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.643062 28600 ts_tablet_manager.cc:1367] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.001s
I20240503 14:12:24.643941 28600 raft_consensus.cc:348] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.644193 28600 raft_consensus.cc:374] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.644251 28600 raft_consensus.cc:729] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.644516 28600 consensus_queue.cc:260] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.645148 28600 ts_tablet_manager.cc:1398] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:24.645164 28614 tablet_bootstrap.cc:492] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.645321 28614 ts_tablet_manager.cc:1367] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.002s
I20240503 14:12:24.645462 28600 tablet_bootstrap.cc:492] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.646056 28614 raft_consensus.cc:348] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.646265 28614 raft_consensus.cc:374] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.646330 28614 raft_consensus.cc:729] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.647341 28600 tablet_bootstrap.cc:654] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.648171 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 117935402a6e4d0d829c446a3de53782 (DEFAULT_TABLE table=default.Table_85137044068a45ce8677074d642ad58b [id=fcaf1fa27ab448d4b62dcc6ba284921f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.648195 28614 consensus_queue.cc:260] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.649323 28614 ts_tablet_manager.cc:1398] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:24.649963 28324 tablet_service.cc:1467] Processing CreateTablet for tablet bfc8ec1d6acb4127b7bf194226135b8c (DEFAULT_TABLE table=default.Table_85a520f4fd804e969bbb4149159976e1 [id=ed81e75ce5ae45a2ac9a547c1903c681]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.650246 28614 tablet_bootstrap.cc:492] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.650725 28334 tablet_service.cc:1467] Processing CreateTablet for tablet e750765798db487fb0b5142f01d21771 (DEFAULT_TABLE table=default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.651382 28323 tablet_service.cc:1467] Processing CreateTablet for tablet b97da778da214bb6bc147c5ada6c7603 (DEFAULT_TABLE table=default.Table_b71e58c4b31248888eecec2d163d0181 [id=e4a9eb2d39d34430bde5d4aae9dccadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.652009 28325 tablet_service.cc:1467] Processing CreateTablet for tablet 7792ad10a98946f68738ff1857edafa3 (DEFAULT_TABLE table=default.Table_865c53e9072b4f2d81c2809a817e35fd [id=0a1aec6101d54a809cde416a450e01eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.648748 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117935402a6e4d0d829c446a3de53782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.653736 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 65e23e136ac94f9fa4180e15a81122a5 (DEFAULT_TABLE table=default.Table_acbb6ef1f5a84c11847e5299afebb8b0 [id=c70dd230e0ac422dbff62e630590ab77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.654222 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e23e136ac94f9fa4180e15a81122a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.655362 28614 tablet_bootstrap.cc:654] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.661128 28187 tablet_service.cc:1467] Processing CreateTablet for tablet ee4afcea52654265afbfe67510d27560 (DEFAULT_TABLE table=default.Table_be835f9c7805488e8c2f84b41440578d [id=38019a96a5554a6bb382a76dea25eda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.661566 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4afcea52654265afbfe67510d27560. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:24.661944 28150 connection.cc:535] client connection to 127.27.50.254:36571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:24.662104 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc8ec1d6acb4127b7bf194226135b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.665751 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e750765798db487fb0b5142f01d21771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.672252 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97da778da214bb6bc147c5ada6c7603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.674088 28614 tablet_bootstrap.cc:492] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:24.674404 28614 ts_tablet_manager.cc:1367] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240503 14:12:24.675330 28614 raft_consensus.cc:348] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.675683 28614 raft_consensus.cc:374] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.675791 28614 raft_consensus.cc:729] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.676278 28325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7792ad10a98946f68738ff1857edafa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.676152 28614 consensus_queue.cc:260] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.678550 28614 ts_tablet_manager.cc:1398] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:24.684934 29750 raft_consensus.cc:482] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.685153 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 844) took 3698 ms. Trace:
I20240503 14:12:24.685127 29750 raft_consensus.cc:504] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.685250 28661 rpcz_store.cc:276] 0503 14:12:20.986644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.987826 (+ 1182us) service_pool.cc:225] Handling call
0503 14:12:24.545138 (+3557312us) ts_tablet_manager.cc:1120] Deleting tablet 7abcb8658fc34c939aa74cf0bd981ae4
0503 14:12:24.545146 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.558611 (+ 13465us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.573001 (+ 14390us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.685127 (+112126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110627,"tablet-delete.queue_time_us":3557062}
I20240503 14:12:24.685628 28661 tablet_replica.cc:330] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.685768 29750 leader_election.cc:288] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.685822 28661 raft_consensus.cc:2229] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:24.685982 28661 pending_rounds.cc:62] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:24.686030 28661 pending_rounds.cc:68] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597741284765696 op_type: NO_OP noop_request { }
I20240503 14:12:24.686121 28661 raft_consensus.cc:2866] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:24.686183 28661 raft_consensus.cc:2258] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.686720 28661 ts_tablet_manager.cc:1875] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.691089 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 27e64d1f0cb341c7ba46def638ffe2ae (DEFAULT_TABLE table=default.Table_c2cb53287e4e4b3781e13698a10d0f43 [id=19f530d8163344159cfd015cf3d6f94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.691505 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e64d1f0cb341c7ba46def638ffe2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.693140 28661 ts_tablet_manager.cc:1888] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:24.693246 28661 log.cc:1196] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/2f89fca0480d4bf28160384ae0421bce
I20240503 14:12:24.693560 28661 ts_tablet_manager.cc:1909] T 2f89fca0480d4bf28160384ae0421bce P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.694370 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 843) took 3778 ms. Trace:
I20240503 14:12:24.694449 28661 rpcz_store.cc:276] 0503 14:12:20.986461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:20.992743 (+ 6282us) service_pool.cc:225] Handling call
0503 14:12:24.685574 (+3692831us) ts_tablet_manager.cc:1120] Deleting tablet 2f89fca0480d4bf28160384ae0421bce
0503 14:12:24.685585 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.689905 (+ 4320us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.693032 (+ 3127us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.694342 (+ 1310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3692349}
I20240503 14:12:24.694769 28661 tablet_replica.cc:330] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.694952 28661 raft_consensus.cc:2229] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.695073 28661 raft_consensus.cc:2258] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.695539 28661 ts_tablet_manager.cc:1875] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.696882 28604 tablet_bootstrap.cc:492] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.697019 28604 ts_tablet_manager.cc:1367] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.198s user 0.053s sys 0.069s
I20240503 14:12:24.698030 28604 raft_consensus.cc:348] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.698239 28604 raft_consensus.cc:374] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.698302 28604 raft_consensus.cc:729] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.698494 28604 consensus_queue.cc:260] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.699406 28604 ts_tablet_manager.cc:1398] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:24.699694 28604 tablet_bootstrap.cc:492] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.705595 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet abb9279e91124fe6bfbdc8856eccfed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 79a96d73b2d142f0b6ada0c1c1cc654e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.706347 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 27e64d1f0cb341c7ba46def638ffe2ae (DEFAULT_TABLE table=default.Table_c2cb53287e4e4b3781e13698a10d0f43 [id=19f530d8163344159cfd015cf3d6f94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.708107 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 2331aec871824a238f3f64de217d7092 (table default.Table_9d9d27627b064475a2f86b5ddcf4236f [id=2d1971a7d562426886d85395dfa1b299]) successfully deleted
I20240503 14:12:24.708366 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 5795755764d9420986aeb10d46f44c45 (DEFAULT_TABLE table=default.Table_1ef3157c088942e4ab395d7e11d0a29d [id=e52e20841e3e41a3b9f311ced4275356]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:24.708504 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet be238ee1d82548c9a1940e2a72e395c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be238ee1d82548c9a1940e2a72e395c7 already in progress: creating tablet
I20240503 14:12:24.708856 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5795755764d9420986aeb10d46f44c45. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:24.708863 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6615052ddbde487e840b6fd77a140e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6615052ddbde487e840b6fd77a140e58 already in progress: creating tablet
W20240503 14:12:24.709127 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f5bea5960cbb44d58d1dbabc888903c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5bea5960cbb44d58d1dbabc888903c2 already in progress: creating tablet
W20240503 14:12:24.709363 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b86c2df0470a4945b3e8207f2d965de2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b86c2df0470a4945b3e8207f2d965de2 already in progress: creating tablet
W20240503 14:12:24.709612 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a31555fc800142bd849dbd572a194002 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a31555fc800142bd849dbd572a194002 already in progress: creating tablet
W20240503 14:12:24.709885 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet dcd405fd84d44076b1792b7fcbe00e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcd405fd84d44076b1792b7fcbe00e84 already in progress: creating tablet
I20240503 14:12:24.710470 28336 tablet_service.cc:1467] Processing CreateTablet for tablet ed0f4a763f0c40479226f0d11323e141 (DEFAULT_TABLE table=default.Table_023451ab297b451da33176a3d13fe295 [id=396cea538c1141618c7cda8c79e3c1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.711093 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 312857b304e04b18b47a4ed6980a5cbf (DEFAULT_TABLE table=default.Table_5fb312c322b34390a3823a7d55e21ebe [id=443266ed484a40c6b15c0098baaeb1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.711689 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 464f9dd19dfb4447b5faebc974671601 (DEFAULT_TABLE table=default.Table_d0f835c02d5f4ffea64369bb8db62de9 [id=fb7f675798544e498af840f47b553dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.712106 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410e8b6b511403ea609b1353855937b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.712414 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 64cf9eee3fc04b2fa4eaa0f92e5a9963 (DEFAULT_TABLE table=default.Table_e0e6e0f2ed4d4e67aef27d3d4337db9b [id=3228cd365fac46c4a8b04f04def9bc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.712512 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet bf21f6234dd64cf8a32f1599fe50746e with delete_type TABLET_DATA_DELETED (replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.712807 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet c5297664a4d943c28adfb6ac4f061680 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:06 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.712965 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba3f01dc93845c5aff20a58178ab9dd (DEFAULT_TABLE table=default.Table_7cd2dac6769a49629f32f98471b10966 [id=031874ef601c4b64bbcb075417207f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.713095 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221218847554cbd9d3242a9a6431e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.713392 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.713661 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 495190fca06a40a386c8215eb2195920 with delete_type TABLET_DATA_DELETED (replaced by tablet 7261088abbc641db8d737a48bbac25bd) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:24.714090 28321 tablet_service.cc:1467] Processing CreateTablet for tablet e8ee387133b6459e9e4730793c92fef8 (DEFAULT_TABLE table=default.Table_eeead1b7b6d54b15845ec251e33f6ca8 [id=ac8ee6ed937244b8b732204c0736a2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.714596 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 (DEFAULT_TABLE table=default.Table_8220e8ba6ccb4f89ba7481c40f8af3ed [id=dd61cf2d027144bf91d38d396333958c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.714990 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 74d5aaf39f1c4a5bb3f9f0c9773b0489 (DEFAULT_TABLE table=default.Table_2746fc24790d430ea83a7a159d11260e [id=243cc5cc92924099b9ea3e7e55c10382]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.715538 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64cf9eee3fc04b2fa4eaa0f92e5a9963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.715939 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 65e23e136ac94f9fa4180e15a81122a5 (DEFAULT_TABLE table=default.Table_acbb6ef1f5a84c11847e5299afebb8b0 [id=c70dd230e0ac422dbff62e630590ab77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.716159 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba3f01dc93845c5aff20a58178ab9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.716465 28329 tablet_service.cc:1467] Processing CreateTablet for tablet ee4afcea52654265afbfe67510d27560 (DEFAULT_TABLE table=default.Table_be835f9c7805488e8c2f84b41440578d [id=38019a96a5554a6bb382a76dea25eda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.716688 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d5aaf39f1c4a5bb3f9f0c9773b0489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.716961 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 5685a6b3ea6943f8a72d74de87ac2166 (DEFAULT_TABLE table=default.Table_d8f7551fc7ad4ea28e5a730ab6f2d0d4 [id=b68e65e25aa74170bdbabb97516414fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.717411 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 74d5aaf39f1c4a5bb3f9f0c9773b0489 (DEFAULT_TABLE table=default.Table_2746fc24790d430ea83a7a159d11260e [id=243cc5cc92924099b9ea3e7e55c10382]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.717854 28335 tablet_service.cc:1467] Processing CreateTablet for tablet addc2a6195034b0a927fdee7a2955122 (DEFAULT_TABLE table=default.Table_d97ddf0257014cd7835a63e1730e6907 [id=9c06c541eab341558ee262552c52b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.718297 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 5795755764d9420986aeb10d46f44c45 (DEFAULT_TABLE table=default.Table_1ef3157c088942e4ab395d7e11d0a29d [id=e52e20841e3e41a3b9f311ced4275356]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.718784 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 64cf9eee3fc04b2fa4eaa0f92e5a9963 (DEFAULT_TABLE table=default.Table_e0e6e0f2ed4d4e67aef27d3d4337db9b [id=3228cd365fac46c4a8b04f04def9bc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.718946 28319 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba3f01dc93845c5aff20a58178ab9dd (DEFAULT_TABLE table=default.Table_7cd2dac6769a49629f32f98471b10966 [id=031874ef601c4b64bbcb075417207f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.720541 28491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe3affd0d144b1fb0078e7690b9b36d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:24.720755 28491 raft_consensus.cc:3037] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.723439 28491 raft_consensus.cc:2454] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:24.728967 28472 tablet_service.cc:1467] Processing CreateTablet for tablet ed0f4a763f0c40479226f0d11323e141 (DEFAULT_TABLE table=default.Table_023451ab297b451da33176a3d13fe295 [id=396cea538c1141618c7cda8c79e3c1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.729650 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0f4a763f0c40479226f0d11323e141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.739078 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 464f9dd19dfb4447b5faebc974671601 (DEFAULT_TABLE table=default.Table_d0f835c02d5f4ffea64369bb8db62de9 [id=fb7f675798544e498af840f47b553dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.739454 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 74d5aaf39f1c4a5bb3f9f0c9773b0489 (DEFAULT_TABLE table=default.Table_2746fc24790d430ea83a7a159d11260e [id=243cc5cc92924099b9ea3e7e55c10382]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.739570 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f9dd19dfb4447b5faebc974671601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.739871 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d5aaf39f1c4a5bb3f9f0c9773b0489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.740162 28456 tablet_service.cc:1467] Processing CreateTablet for tablet e8ee387133b6459e9e4730793c92fef8 (DEFAULT_TABLE table=default.Table_eeead1b7b6d54b15845ec251e33f6ca8 [id=ac8ee6ed937244b8b732204c0736a2c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.740509 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ee387133b6459e9e4730793c92fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.744416 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 (DEFAULT_TABLE table=default.Table_8220e8ba6ccb4f89ba7481c40f8af3ed [id=dd61cf2d027144bf91d38d396333958c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.744817 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5087cf4a795f4ee08ce4dd19d095d0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.744832 28180 tablet_service.cc:1467] Processing CreateTablet for tablet addc2a6195034b0a927fdee7a2955122 (DEFAULT_TABLE table=default.Table_d97ddf0257014cd7835a63e1730e6907 [id=9c06c541eab341558ee262552c52b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.745203 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addc2a6195034b0a927fdee7a2955122. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.748448 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 65e23e136ac94f9fa4180e15a81122a5 (DEFAULT_TABLE table=default.Table_acbb6ef1f5a84c11847e5299afebb8b0 [id=c70dd230e0ac422dbff62e630590ab77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.748895 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e23e136ac94f9fa4180e15a81122a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.748862 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 5685a6b3ea6943f8a72d74de87ac2166 (DEFAULT_TABLE table=default.Table_d8f7551fc7ad4ea28e5a730ab6f2d0d4 [id=b68e65e25aa74170bdbabb97516414fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.750424 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5685a6b3ea6943f8a72d74de87ac2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.751724 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bba6a3d6e84552b027f23a29cbb38a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:24.751878 28469 tablet_service.cc:1467] Processing CreateTablet for tablet addc2a6195034b0a927fdee7a2955122 (DEFAULT_TABLE table=default.Table_d97ddf0257014cd7835a63e1730e6907 [id=9c06c541eab341558ee262552c52b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.752372 28470 tablet_service.cc:1467] Processing CreateTablet for tablet ee4afcea52654265afbfe67510d27560 (DEFAULT_TABLE table=default.Table_be835f9c7805488e8c2f84b41440578d [id=38019a96a5554a6bb382a76dea25eda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.752496 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addc2a6195034b0a927fdee7a2955122. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.752743 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4afcea52654265afbfe67510d27560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.754051 28604 tablet_bootstrap.cc:654] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.756446 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 5685a6b3ea6943f8a72d74de87ac2166 (DEFAULT_TABLE table=default.Table_d8f7551fc7ad4ea28e5a730ab6f2d0d4 [id=b68e65e25aa74170bdbabb97516414fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.756860 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5685a6b3ea6943f8a72d74de87ac2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.760529 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 5795755764d9420986aeb10d46f44c45 (DEFAULT_TABLE table=default.Table_1ef3157c088942e4ab395d7e11d0a29d [id=e52e20841e3e41a3b9f311ced4275356]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.760942 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5795755764d9420986aeb10d46f44c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.761912 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fad4508a29d4822b81af6947b57e7ef" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:24.764200 28473 tablet_service.cc:1467] Processing CreateTablet for tablet 64cf9eee3fc04b2fa4eaa0f92e5a9963 (DEFAULT_TABLE table=default.Table_e0e6e0f2ed4d4e67aef27d3d4337db9b [id=3228cd365fac46c4a8b04f04def9bc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.764528 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64cf9eee3fc04b2fa4eaa0f92e5a9963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.766770 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba3f01dc93845c5aff20a58178ab9dd (DEFAULT_TABLE table=default.Table_7cd2dac6769a49629f32f98471b10966 [id=031874ef601c4b64bbcb075417207f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.767055 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba3f01dc93845c5aff20a58178ab9dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:24.772336 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 62399.6us
W20240503 14:12:24.772706 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet cf221f17ddfd461f83f7f916d7833011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf221f17ddfd461f83f7f916d7833011 already in progress: creating tablet
W20240503 14:12:24.772967 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a3d1314841574fc8bcf3ccb16755e421 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3d1314841574fc8bcf3ccb16755e421 already in progress: creating tablet
W20240503 14:12:24.773306 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a53279669bad40ad9243f4554a4e3c70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53279669bad40ad9243f4554a4e3c70 already in progress: creating tablet
W20240503 14:12:24.773499 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 4e11137b75794e2c9dd61ded326c868f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e11137b75794e2c9dd61ded326c868f already in progress: creating tablet
W20240503 14:12:24.773859 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6f0d676fa1e74ab89922f7ba607fd2d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f0d676fa1e74ab89922f7ba607fd2d8 already in progress: creating tablet
I20240503 14:12:24.774056 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 95c97a43bd714da68c1a77408f3d1cd1 (table default.Table_c7035b3f783948498dfb7bd5ed14aad8 [id=15ae3079abdd4069aaeb84e1d85da988]) successfully deleted
W20240503 14:12:24.774298 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1603e2360041427f8635782e812b6431 because the tablet was not found. No further retry: Not found: Tablet not found: 1603e2360041427f8635782e812b6431
I20240503 14:12:24.774631 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7abcb8658fc34c939aa74cf0bd981ae4 (table default.Table_80c5e40f9e494b2dbbbb7449a5b36569 [id=07297b58e7ca4fb1bf0e050a341d2c9c]) successfully deleted
I20240503 14:12:24.774821 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 2f89fca0480d4bf28160384ae0421bce (table default.Table_5654996201694c1b90fc6e728e068ca3 [id=9a6689ece91646069be1867e8f141392]) successfully deleted
I20240503 14:12:24.778683 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 312857b304e04b18b47a4ed6980a5cbf (DEFAULT_TABLE table=default.Table_5fb312c322b34390a3823a7d55e21ebe [id=443266ed484a40c6b15c0098baaeb1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.780671 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312857b304e04b18b47a4ed6980a5cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.785341 28661 ts_tablet_manager.cc:1888] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:24.785482 28661 log.cc:1196] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7b6c998e518541248a87e4c1551ec080
I20240503 14:12:24.785807 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe3affd0d144b1fb0078e7690b9b36d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:24.786033 28213 raft_consensus.cc:3037] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.789033 28213 raft_consensus.cc:2454] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:24.789196 28600 tablet_bootstrap.cc:492] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.789359 28600 ts_tablet_manager.cc:1367] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.144s user 0.025s sys 0.003s
I20240503 14:12:24.790154 28600 raft_consensus.cc:348] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.790366 28600 raft_consensus.cc:374] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.790427 28600 raft_consensus.cc:729] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.790627 28600 consensus_queue.cc:260] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.792445 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbcae22ea124edaa14c9409f3052714" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:24.793968 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e64d1f0cb341c7ba46def638ffe2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.795598 28600 ts_tablet_manager.cc:1398] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:24.795717 29903 consensus_queue.cc:1035] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.796000 28600 tablet_bootstrap.cc:492] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.796638 28614 tablet_bootstrap.cc:492] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:24.797876 28600 tablet_bootstrap.cc:654] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.798338 28614 tablet_bootstrap.cc:654] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.799533 29828 tablet_replica.cc:330] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:24.799737 29828 raft_consensus.cc:2229] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.799898 29828 raft_consensus.cc:2258] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.802285 29828 ts_tablet_manager.cc:1875] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.808003 29903 consensus_queue.cc:1035] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.813413 28661 ts_tablet_manager.cc:1909] T 7b6c998e518541248a87e4c1551ec080 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.815268 28604 tablet_bootstrap.cc:492] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:24.815446 28604 ts_tablet_manager.cc:1367] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.003s
I20240503 14:12:24.817596 29949 raft_consensus.cc:482] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.817749 29949 raft_consensus.cc:504] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.818487 29949 leader_election.cc:288] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:24.830319 29828 ts_tablet_manager.cc:1888] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:24.830610 29828 log.cc:1196] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/8410e8b6b511403ea609b1353855937b
I20240503 14:12:24.831036 29828 ts_tablet_manager.cc:1909] T 8410e8b6b511403ea609b1353855937b P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:24.832191 29828 tablet_replica.cc:330] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:24.832988 29828 raft_consensus.cc:2229] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.833982 29828 raft_consensus.cc:2258] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.836602 28600 tablet_bootstrap.cc:492] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.836797 28600 ts_tablet_manager.cc:1367] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.041s user 0.016s sys 0.024s
I20240503 14:12:24.837644 28600 raft_consensus.cc:348] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.837881 28600 raft_consensus.cc:374] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.837967 28600 raft_consensus.cc:729] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.838228 28600 consensus_queue.cc:260] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.841037 28600 ts_tablet_manager.cc:1398] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:24.841429 28600 tablet_bootstrap.cc:492] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.843477 28600 tablet_bootstrap.cc:654] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.848839 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 847) took 3788 ms. Trace:
I20240503 14:12:24.848966 28661 rpcz_store.cc:276] 0503 14:12:21.060324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.060384 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:24.694718 (+3634334us) ts_tablet_manager.cc:1120] Deleting tablet 7b6c998e518541248a87e4c1551ec080
0503 14:12:24.694729 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.752041 (+ 57312us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.785156 (+ 33115us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.848810 (+ 63654us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34417,"tablet-delete.queue_time_us":3634100}
I20240503 14:12:24.849289 28661 tablet_replica.cc:330] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.849514 28661 raft_consensus.cc:2229] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:24.849686 28661 pending_rounds.cc:62] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:24.849730 28661 pending_rounds.cc:68] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597741044064256 op_type: NO_OP noop_request { }
I20240503 14:12:24.849835 28661 raft_consensus.cc:2866] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:24.849910 28661 raft_consensus.cc:2258] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.850498 28661 ts_tablet_manager.cc:1875] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.852016 29903 consensus_queue.cc:1035] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.852491 28604 raft_consensus.cc:348] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.852720 28604 raft_consensus.cc:374] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.852769 28604 raft_consensus.cc:729] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.852977 28604 consensus_queue.cc:260] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.853547 28604 ts_tablet_manager.cc:1398] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240503 14:12:24.853848 28604 tablet_bootstrap.cc:492] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:24.855705 28604 tablet_bootstrap.cc:654] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.875097 28661 ts_tablet_manager.cc:1888] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:24.879441 28661 log.cc:1196] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:24.879858 28600 tablet_bootstrap.cc:492] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:24.880036 28600 ts_tablet_manager.cc:1367] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.005s
I20240503 14:12:24.880808 28661 ts_tablet_manager.cc:1909] T e0a92fc3833e49aaab8c5415de9bb685 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:24.880928 28600 raft_consensus.cc:348] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.881155 28600 raft_consensus.cc:374] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.881222 28600 raft_consensus.cc:729] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.881470 28600 consensus_queue.cc:260] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.881662 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 848) took 3820 ms. Trace:
I20240503 14:12:24.881762 28661 rpcz_store.cc:276] 0503 14:12:21.060831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.060918 (+ 87us) service_pool.cc:225] Handling call
0503 14:12:24.849240 (+3788322us) ts_tablet_manager.cc:1120] Deleting tablet e0a92fc3833e49aaab8c5415de9bb685
0503 14:12:24.849249 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.859427 (+ 10178us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.874956 (+ 15529us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.881621 (+ 6665us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3788147}
I20240503 14:12:24.882033 28600 ts_tablet_manager.cc:1398] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:24.882105 28661 tablet_replica.cc:330] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:24.882290 28661 raft_consensus.cc:2229] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:24.885031 28600 tablet_bootstrap.cc:492] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:24.886915 28600 tablet_bootstrap.cc:654] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.891477 28661 raft_consensus.cc:2258] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:24.901669 28661 ts_tablet_manager.cc:1875] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.912724 30005 consensus_queue.cc:1035] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.964299 30005 raft_consensus.cc:482] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.964538 30005 raft_consensus.cc:504] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:24.965380 30005 leader_election.cc:288] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:24.888617 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0f4a763f0c40479226f0d11323e141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.976266 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addc2a6195034b0a927fdee7a2955122. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.980376 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4afcea52654265afbfe67510d27560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.986575 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5087cf4a795f4ee08ce4dd19d095d0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.991887 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba3f01dc93845c5aff20a58178ab9dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.996029 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d5aaf39f1c4a5bb3f9f0c9773b0489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.003049 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ee387133b6459e9e4730793c92fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.007561 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64cf9eee3fc04b2fa4eaa0f92e5a9963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.011901 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5795755764d9420986aeb10d46f44c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.015842 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5685a6b3ea6943f8a72d74de87ac2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.019841 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f9dd19dfb4447b5faebc974671601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.025367 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e23e136ac94f9fa4180e15a81122a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.028281 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312857b304e04b18b47a4ed6980a5cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.954370 28661 ts_tablet_manager.cc:1888] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.040347 28661 log.cc:1196] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/c5297664a4d943c28adfb6ac4f061680
I20240503 14:12:25.040709 28661 ts_tablet_manager.cc:1909] T c5297664a4d943c28adfb6ac4f061680 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:25.041651 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 849) took 3980 ms. Trace:
I20240503 14:12:25.041766 28661 rpcz_store.cc:276] 0503 14:12:21.061450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.061520 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:24.882052 (+3820532us) ts_tablet_manager.cc:1120] Deleting tablet c5297664a4d943c28adfb6ac4f061680
0503 14:12:24.882063 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:24.928006 (+ 45943us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:24.953460 (+ 25454us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.041616 (+ 88156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3820340}
I20240503 14:12:25.042129 28661 tablet_replica.cc:330] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:25.042325 28661 raft_consensus.cc:2229] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:25.042474 28661 raft_consensus.cc:2258] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.043078 28661 ts_tablet_manager.cc:1875] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:24.966835 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75036cdf9e04ea4aa767fd12016f177" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.006021 29750 raft_consensus.cc:482] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.044847 29750 raft_consensus.cc:504] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.045481 29750 leader_election.cc:288] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:25.045928 28150 leader_election.cc:341] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.052310 28661 ts_tablet_manager.cc:1888] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.052481 28661 log.cc:1196] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/8410e8b6b511403ea609b1353855937b
I20240503 14:12:25.052846 28661 ts_tablet_manager.cc:1909] T 8410e8b6b511403ea609b1353855937b P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:25.053747 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 846) took 3993 ms. Trace:
I20240503 14:12:25.053854 28661 rpcz_store.cc:276] 0503 14:12:21.060111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.069380 (+ 9269us) service_pool.cc:225] Handling call
0503 14:12:25.042070 (+3972690us) ts_tablet_manager.cc:1120] Deleting tablet 8410e8b6b511403ea609b1353855937b
0503 14:12:25.042081 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.048611 (+ 6530us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.052163 (+ 3552us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.053714 (+ 1551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3972292}
I20240503 14:12:25.054251 28661 tablet_replica.cc:330] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:25.054468 28661 raft_consensus.cc:2229] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:25.054625 28661 raft_consensus.cc:2258] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.055241 28661 ts_tablet_manager.cc:1875] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.061842 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137ef4cfc2476cbc70f1cc72ad89e6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.062116 28493 raft_consensus.cc:2454] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:25.062974 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137ef4cfc2476cbc70f1cc72ad89e6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.063238 28207 raft_consensus.cc:2454] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
W20240503 14:12:24.877280 28427 connection.cc:535] client connection to 127.27.50.254:36571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:25.067142 29828 ts_tablet_manager.cc:1875] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.087293 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75036cdf9e04ea4aa767fd12016f177" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.087476 28346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fad4508a29d4822b81af6947b57e7ef" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.087709 28346 raft_consensus.cc:2379] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:25.087811 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bba6a3d6e84552b027f23a29cbb38a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.088152 28351 raft_consensus.cc:2379] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
W20240503 14:12:25.090224 28150 leader_election.cc:341] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.090428 28150 leader_election.cc:302] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.090704 30005 raft_consensus.cc:2726] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.091159 29811 consensus_queue.cc:1035] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.087208 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920be6d87c4440d0bf7453c86cfe0b5c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.107831 28614 tablet_bootstrap.cc:492] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
W20240503 14:12:25.141451 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 365325us
I20240503 14:12:25.108572 30097 tablet.cc:1722] T 240abf8c2b2343408fc98a8dac1fdf09 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:25.145292 28614 ts_tablet_manager.cc:1367] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.349s user 0.008s sys 0.004s
I20240503 14:12:25.161527 28491 raft_consensus.cc:3037] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.170164 28604 tablet_bootstrap.cc:492] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.170358 28604 ts_tablet_manager.cc:1367] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.317s user 0.011s sys 0.047s
I20240503 14:12:25.171203 28604 raft_consensus.cc:348] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.171428 28604 raft_consensus.cc:374] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.171490 28604 raft_consensus.cc:729] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.171715 28604 consensus_queue.cc:260] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.172303 28604 ts_tablet_manager.cc:1398] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:25.172623 28604 tablet_bootstrap.cc:492] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.174719 28604 tablet_bootstrap.cc:654] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.175102 27851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
/tmp/dist-test-taskwmRrLf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.254:36571
--webserver_interface=127.27.50.254
--webserver_port=38435
--builtin_ntp_servers=127.27.50.212:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240503 14:12:25.175437 28489 raft_consensus.cc:3037] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.179503 29903 consensus_queue.cc:1035] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.179489 28614 raft_consensus.cc:348] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.179726 28614 raft_consensus.cc:374] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.179809 28614 raft_consensus.cc:729] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.180047 28614 consensus_queue.cc:260] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:25.181190 28427 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 104681us
I20240503 14:12:25.193955 29750 consensus_queue.cc:1035] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:25.194836 28427 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 903f11a608b1483887d9b5234a404470
I20240503 14:12:25.195590 30056 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 14:12:25.198971 28604 tablet_bootstrap.cc:492] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.199242 28604 ts_tablet_manager.cc:1367] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.001s
I20240503 14:12:25.200253 28604 raft_consensus.cc:348] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.200551 28604 raft_consensus.cc:374] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.200649 28604 raft_consensus.cc:729] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.200901 28604 consensus_queue.cc:260] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.201516 28604 ts_tablet_manager.cc:1398] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:25.201896 28604 tablet_bootstrap.cc:492] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.206395 28030 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1102) took 1253 ms. Trace:
I20240503 14:12:25.206523 28030 rpcz_store.cc:276] 0503 14:12:23.953078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.479852 (+526774us) service_pool.cc:225] Handling call
0503 14:12:24.498988 (+ 19136us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:24.499975 (+ 987us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:24.500475 (+ 500us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.206111 (+705636us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.206251 (+ 140us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.206368 (+ 117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:24.505460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.505583 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.505591 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:24.505593 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:24.505694 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.505708 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.505713 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:24.505800 (+ 87us) tablet.cc:699] Row locks acquired
0503 14:12:24.505803 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:24.505840 (+ 37us) write_op.cc:270] Start()
0503 14:12:24.505875 (+ 35us) write_op.cc:276] Timestamp: P: 1714745544505820 usec, L: 0
0503 14:12:24.505878 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.547892 (+ 42014us) log.cc:842] Serialized 433 byte log entry
0503 14:12:24.548535 (+ 643us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 000000000095519a 00007fe125139162 00007fe1264a1e32 00007fe1264a0959 00007fe1264a886b 00007fe1264a86e8 0000000000a60bcc 00007fe11993ab74 00007fe11993e19b 00007fe11993e018 0000000000a60bcc
0503 14:12:25.202200 (+653665us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.204375 (+ 2175us) write_op.cc:301] APPLY: starting
0503 14:12:25.204399 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.204454 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.204457 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.204594 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.204638 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.204646 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.204770 (+ 124us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.205810 (+ 1040us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.205838 (+ 28us) write_op.cc:454] Released schema lock
0503 14:12:25.205849 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":43319168,"child_traces":[["op",{"apply.queue_time_us":2079,"num_ops":2,"prepare.queue_time_us":4446,"prepare.run_cpu_time_us":1006,"prepare.run_wall_time_us":43138,"raft.queue_time_us":1225,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":256,"replication_time_us":696265,"spinlock_wait_cycles":96197376,"threads_started":1}]]}
I20240503 14:12:25.218444 28614 ts_tablet_manager.cc:1398] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.073s user 0.002s sys 0.000s
I20240503 14:12:25.218839 28614 tablet_bootstrap.cc:492] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:25.223376 28032 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1101) took 1270 ms. Trace:
I20240503 14:12:25.223522 28032 rpcz_store.cc:276] 0503 14:12:23.952928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.479175 (+526247us) service_pool.cc:225] Handling call
0503 14:12:24.501508 (+ 22333us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:24.502369 (+ 861us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:24.502775 (+ 406us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.223075 (+720300us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.223217 (+ 142us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.223343 (+ 126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:24.548637 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.548757 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.548764 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:24.548766 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:24.548859 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.548872 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.548877 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:24.548974 (+ 97us) tablet.cc:699] Row locks acquired
0503 14:12:24.548977 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:24.549023 (+ 46us) write_op.cc:270] Start()
0503 14:12:24.549060 (+ 37us) write_op.cc:276] Timestamp: P: 1714745544548996 usec, L: 0
0503 14:12:24.549065 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.549226 (+ 161us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.220179 (+670953us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.222177 (+ 1998us) write_op.cc:301] APPLY: starting
0503 14:12:25.222200 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.222259 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.222294 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.222423 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.222470 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.222478 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.222584 (+ 106us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.222937 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.222966 (+ 29us) write_op.cc:454] Released schema lock
0503 14:12:25.222976 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":50629376,"child_traces":[["op",{"apply.queue_time_us":39,"num_ops":2,"prepare.queue_time_us":45398,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":1507,"raft.queue_time_us":338,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":671067}]]}
W20240503 14:12:25.228161 28290 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 163475us
I20240503 14:12:25.232766 29996 raft_consensus.cc:482] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.233372 29996 raft_consensus.cc:504] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.235230 29996 leader_election.cc:288] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:25.236248 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 53760.5us
I20240503 14:12:25.237157 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 41cbd1326869434594e126421ad132d8 (table default.Table_a07e2f0650084270860bd28e40f224aa [id=a98b2a6e32bf425eba5d2e039def6b0f]) successfully deleted
W20240503 14:12:25.238858 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 00a56d79c36f4ffa9bae02e2369d8ded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00a56d79c36f4ffa9bae02e2369d8ded already in progress: creating tablet
I20240503 14:12:25.239740 28604 tablet_bootstrap.cc:654] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:25.242403 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 380f267a984541c7878eeaa4eeee8f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 380f267a984541c7878eeaa4eeee8f34 already in progress: creating tablet
W20240503 14:12:25.242694 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0c300b1f104949b3ae74f5b38d378a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c300b1f104949b3ae74f5b38d378a11 already in progress: creating tablet
W20240503 14:12:25.242892 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 86804987bdf340dc83ddd58ba2bc8a96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86804987bdf340dc83ddd58ba2bc8a96 already in progress: creating tablet
I20240503 14:12:25.243072 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ed4dc732bcb44222b4b15e5d77defe33 (table default.Table_3ce7004b7c164629a92e6bcbb421348d [id=a6a9ad0f171448c2a3d3401e02ec62c0]) successfully deleted
W20240503 14:12:25.243258 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 11e14e8b5f4046029af4673a277112d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11e14e8b5f4046029af4673a277112d9 already in progress: creating tablet
W20240503 14:12:25.243436 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 already in progress: creating tablet
W20240503 14:12:25.243619 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 1132014802ed4c3682769ba86312eed2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1132014802ed4c3682769ba86312eed2 already in progress: creating tablet
W20240503 14:12:25.243789 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d21bdd6490004300bca7a5a7fb24cd07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d21bdd6490004300bca7a5a7fb24cd07 already in progress: creating tablet
W20240503 14:12:25.241633 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 46496.3us
I20240503 14:12:25.245427 28290 leader_election.cc:302] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.246788 28427 leader_election.cc:302] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:25.247475 28290 consensus_peers.cc:485] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5297664a4d943c28adfb6ac4f061680. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:25.248816 29811 consensus_queue.cc:1035] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.253134 28600 tablet_bootstrap.cc:492] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.253448 28600 ts_tablet_manager.cc:1367] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.369s user 0.100s sys 0.127s
I20240503 14:12:25.254354 28600 raft_consensus.cc:348] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.254681 28600 raft_consensus.cc:374] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.254786 28600 raft_consensus.cc:729] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.255069 28600 consensus_queue.cc:260] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.255496 29750 raft_consensus.cc:3037] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.255734 28600 ts_tablet_manager.cc:1398] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:25.256038 28600 tablet_bootstrap.cc:492] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.257748 28600 tablet_bootstrap.cc:654] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.258167 29750 raft_consensus.cc:2726] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.259968 28290 leader_election.cc:302] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.261582 30005 raft_consensus.cc:482] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.261744 30005 raft_consensus.cc:504] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.262391 30005 leader_election.cc:288] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:25.262531 29949 raft_consensus.cc:2781] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.262647 29949 raft_consensus.cc:482] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.262768 29949 raft_consensus.cc:3037] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.264145 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a544fc27624dcc8a4cb003e59408f2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:25.265760 28150 leader_election.cc:341] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.265951 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a544fc27624dcc8a4cb003e59408f2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:25.266978 28150 leader_election.cc:341] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.267104 28150 leader_election.cc:302] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.267253 29949 raft_consensus.cc:504] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.267688 29811 raft_consensus.cc:2726] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.267992 29949 leader_election.cc:288] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:25.269415 28427 consensus_peers.cc:485] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed4dc732bcb44222b4b15e5d77defe33. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:25.274214 28604 tablet_bootstrap.cc:492] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.274549 28604 ts_tablet_manager.cc:1367] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20240503 14:12:25.274804 28614 tablet_bootstrap.cc:654] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.275851 28604 raft_consensus.cc:348] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.276202 28604 raft_consensus.cc:374] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.276324 28604 raft_consensus.cc:729] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.276624 28604 consensus_queue.cc:260] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.277268 28604 ts_tablet_manager.cc:1398] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:25.277325 28600 tablet_bootstrap.cc:492] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.254073 30107 raft_consensus.cc:2781] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.277606 28600 ts_tablet_manager.cc:1367] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 14:12:25.277626 30107 raft_consensus.cc:686] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:25.278052 30107 consensus_queue.cc:237] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.278510 28600 raft_consensus.cc:348] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.278825 28600 raft_consensus.cc:374] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.278928 28600 raft_consensus.cc:729] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.279245 28600 consensus_queue.cc:260] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.279403 29828 ts_tablet_manager.cc:1888] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.279868 28600 ts_tablet_manager.cc:1398] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:25.280551 30070 consensus_queue.cc:1035] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.281412 28600 tablet_bootstrap.cc:492] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:25.282927 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet abb9279e91124fe6bfbdc8856eccfed8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abb9279e91124fe6bfbdc8856eccfed8 already in progress: creating tablet
I20240503 14:12:25.283035 28600 tablet_bootstrap.cc:654] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.283356 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 8410e8b6b511403ea609b1353855937b (table default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280 [id=05881d1136974066b34909e625c970a7]) successfully deleted
W20240503 14:12:25.283648 28017 connection.cc:535] client connection to 127.27.50.254:36571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:25.283802 28017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20240503 14:12:25.292379 29949 raft_consensus.cc:482] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.292775 29949 raft_consensus.cc:504] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.293653 29949 leader_election.cc:288] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.295238 30106 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:12:25.295523 30106 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
I20240503 14:12:25.299688 28600 tablet_bootstrap.cc:492] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.299964 28600 ts_tablet_manager.cc:1367] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20240503 14:12:25.300854 28600 raft_consensus.cc:348] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.301163 28600 raft_consensus.cc:374] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.301265 28600 raft_consensus.cc:729] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.301558 28600 consensus_queue.cc:260] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.302208 28600 ts_tablet_manager.cc:1398] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:25.302592 28600 tablet_bootstrap.cc:492] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.304443 28600 tablet_bootstrap.cc:654] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.313162 29949 raft_consensus.cc:482] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.313510 29949 raft_consensus.cc:504] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.314407 29949 leader_election.cc:288] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.305207 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.306082 29811 raft_consensus.cc:482] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.316412 29811 raft_consensus.cc:504] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.317221 29811 leader_election.cc:288] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.308071 29903 raft_consensus.cc:482] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.317447 29903 raft_consensus.cc:504] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.318078 29903 leader_election.cc:288] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.318151 28614 tablet_bootstrap.cc:492] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:25.318276 28212 raft_consensus.cc:2454] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:25.318464 28614 ts_tablet_manager.cc:1367] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.002s
I20240503 14:12:25.319399 28614 raft_consensus.cc:348] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.319823 28614 raft_consensus.cc:374] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.319945 28614 raft_consensus.cc:729] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.320894 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1958dc488e4854a8c8e74da9fab243" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.321133 28357 raft_consensus.cc:2454] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.320737 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.321573 28150 leader_election.cc:302] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.321861 29903 raft_consensus.cc:2781] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.321952 29903 raft_consensus.cc:482] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.322023 29903 raft_consensus.cc:3037] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.325096 29903 raft_consensus.cc:504] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.325876 29903 leader_election.cc:288] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:25.327030 28150 leader_election.cc:341] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.327200 28600 tablet_bootstrap.cc:492] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.327558 28600 ts_tablet_manager.cc:1367] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240503 14:12:25.327710 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1958dc488e4854a8c8e74da9fab243" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.327955 28354 raft_consensus.cc:3037] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.328608 28600 raft_consensus.cc:348] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.328927 28600 raft_consensus.cc:374] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.329026 28600 raft_consensus.cc:729] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.329340 28600 consensus_queue.cc:260] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.329926 28600 ts_tablet_manager.cc:1398] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:25.330626 28600 tablet_bootstrap.cc:492] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.331696 28354 raft_consensus.cc:2454] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:25.332486 28150 leader_election.cc:302] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.332859 29903 raft_consensus.cc:2781] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.333294 28600 tablet_bootstrap.cc:654] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.333542 29903 raft_consensus.cc:686] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:25.333916 29903 consensus_queue.cc:237] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.320598 28614 consensus_queue.cc:260] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.339903 28614 ts_tablet_manager.cc:1398] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240503 14:12:25.340340 28614 tablet_bootstrap.cc:492] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:25.342442 28614 tablet_bootstrap.cc:654] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.344213 30005 consensus_queue.cc:1035] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:12:25.347615 30005 raft_consensus.cc:482] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.347796 30005 raft_consensus.cc:504] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.348526 30005 leader_election.cc:288] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.351359 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd405fd84d44076b1792b7fcbe00e84" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.351615 28354 raft_consensus.cc:2454] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.353618 28150 leader_election.cc:302] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.354020 30005 raft_consensus.cc:2781] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.354182 30005 raft_consensus.cc:482] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.354310 30005 raft_consensus.cc:3037] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.356915 30005 raft_consensus.cc:504] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.357704 30005 leader_election.cc:288] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.361752 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd405fd84d44076b1792b7fcbe00e84" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.362000 28354 raft_consensus.cc:3037] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.364120 29828 log.cc:1196] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/bf21f6234dd64cf8a32f1599fe50746e
I20240503 14:12:25.364630 29828 ts_tablet_manager.cc:1909] T bf21f6234dd64cf8a32f1599fe50746e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:25.364737 28354 raft_consensus.cc:2454] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:25.365128 28600 tablet_bootstrap.cc:492] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.365422 28600 ts_tablet_manager.cc:1367] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.003s
I20240503 14:12:25.365556 29828 tablet_replica.cc:330] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:25.365839 29828 raft_consensus.cc:2229] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:25.366293 29828 pending_rounds.cc:62] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:25.366294 28600 raft_consensus.cc:348] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.366423 29828 pending_rounds.cc:68] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597748154617856 op_type: NO_OP noop_request { }
I20240503 14:12:25.366577 29828 raft_consensus.cc:2866] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:25.366627 28600 raft_consensus.cc:374] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.366736 29828 raft_consensus.cc:2258] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.366788 28600 raft_consensus.cc:729] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.367079 28600 consensus_queue.cc:260] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.367422 29828 ts_tablet_manager.cc:1875] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.368176 28600 ts_tablet_manager.cc:1398] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:25.369072 28150 leader_election.cc:302] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.369581 28600 tablet_bootstrap.cc:492] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.369840 30005 raft_consensus.cc:2781] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.370105 30005 raft_consensus.cc:686] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:25.370455 30005 consensus_queue.cc:237] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.371336 28600 tablet_bootstrap.cc:654] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.395463 30121 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
W20240503 14:12:25.399123 29436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111)
I20240503 14:12:25.399678 28600 tablet_bootstrap.cc:492] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.399870 28600 ts_tablet_manager.cc:1367] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.007s
I20240503 14:12:25.400702 28600 raft_consensus.cc:348] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.400903 28600 raft_consensus.cc:374] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.400947 28600 raft_consensus.cc:729] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.401160 28600 consensus_queue.cc:260] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.405731 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1168 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:12:25.406719 30106 leader_election.cc:288] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
W20240503 14:12:25.415879 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 132065us
I20240503 14:12:25.417263 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7b6c998e518541248a87e4c1551ec080 (table default.Table_a7aefabd0c3841618e68382b8fa41a9a [id=c23a0b9e98f949559497aea2a4dab24a]) successfully deleted
I20240503 14:12:25.417507 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e0a92fc3833e49aaab8c5415de9bb685 (table default.Table_9953322d44f34ce5ad1123ba2784b66f [id=5ae48705fb644bddba8e69d66c640a68]) successfully deleted
I20240503 14:12:25.417647 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet c5297664a4d943c28adfb6ac4f061680 (table default.Table_78145a3b2c8543b5ab5a675ab4b08c91 [id=99ed67dc285f4dc69a9a9c6cec10b729]) successfully deleted
I20240503 14:12:25.417779 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 8410e8b6b511403ea609b1353855937b (table default.Table_e0e6f362b6bb44ec9dd98d8b0c9e8280 [id=05881d1136974066b34909e625c970a7]) successfully deleted
I20240503 14:12:25.303192 29996 raft_consensus.cc:482] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.319557 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.431166 28493 raft_consensus.cc:2454] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.431160 29996 raft_consensus.cc:504] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.431908 29996 leader_election.cc:288] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.320343 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1958dc488e4854a8c8e74da9fab243" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.432204 28486 raft_consensus.cc:2454] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.326938 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1958dc488e4854a8c8e74da9fab243" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:25.432453 28488 raft_consensus.cc:3037] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.433566 28600 ts_tablet_manager.cc:1398] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.034s user 0.002s sys 0.007s
I20240503 14:12:25.433912 28600 tablet_bootstrap.cc:492] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.435213 28488 raft_consensus.cc:2454] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:25.334568 28427 leader_election.cc:302] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:25.435577 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 101042us
I20240503 14:12:25.435715 28600 tablet_bootstrap.cc:654] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.436023 29750 raft_consensus.cc:2781] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.436133 29750 raft_consensus.cc:482] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.338663 28661 ts_tablet_manager.cc:1888] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.436209 29750 raft_consensus.cc:3037] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.436311 28661 log.cc:1196] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/bf21f6234dd64cf8a32f1599fe50746e
I20240503 14:12:25.436645 28661 ts_tablet_manager.cc:1909] T bf21f6234dd64cf8a32f1599fe50746e P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:25.437507 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 845) took 4377 ms. Trace:
I20240503 14:12:25.437598 28661 rpcz_store.cc:276] 0503 14:12:21.059824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.070363 (+ 10539us) service_pool.cc:225] Handling call
0503 14:12:25.054191 (+3983828us) ts_tablet_manager.cc:1120] Deleting tablet bf21f6234dd64cf8a32f1599fe50746e
0503 14:12:25.054205 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.294767 (+240562us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.338516 (+ 43749us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.437482 (+ 98966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3983508}
I20240503 14:12:25.438004 28661 tablet_replica.cc:330] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:25.438192 28661 raft_consensus.cc:2229] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:25.438339 28661 raft_consensus.cc:2258] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.438905 28661 ts_tablet_manager.cc:1875] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.439085 29750 raft_consensus.cc:504] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:25.439488 29436 leader_election.cc:334] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571): Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111)
I20240503 14:12:25.439677 29436 leader_election.cc:302] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc; no voters: bb6a7aee99214df4b02858ddae025b14, c0690a516c89446d85c88cdf64fa4238
I20240503 14:12:25.439819 29750 leader_election.cc:288] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.351501 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd405fd84d44076b1792b7fcbe00e84" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.440467 30106 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:25.440527 28487 raft_consensus.cc:2454] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.360852 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd405fd84d44076b1792b7fcbe00e84" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:25.440804 28494 raft_consensus.cc:3037] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.443384 28494 raft_consensus.cc:2454] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
W20240503 14:12:25.447209 28017 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet b0993be031cc4792a9eadba9ce41d024 (table default.Table_4ebf150e25b946eab62caf24e5a3c292 [id=892862b894c547e384ebf8b33b9a7ba5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b0993be031cc4792a9eadba9ce41d024 consensus state has no leader
W20240503 14:12:25.447522 28017 consensus_peers.cc:485] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 -> Peer bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571): Couldn't send request to peer bb6a7aee99214df4b02858ddae025b14. Status: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:25.455935 28030 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet e94673f00cb3432185009bda2ce398fa (table default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d [id=2849f6ba9ef64d26a5dc65f9ac9048da]). Expected version 1 got 0
I20240503 14:12:25.456250 28030 catalog_manager.cc:5484] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:25.457206 28030 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 77c77a11ef384709aef5b62273380573 (table default.Table_ab495b10ad82425ba5c7123a6e1cf064 [id=2edd1164c07a4a67b51262d402d53596]). Expected version 2 got 0
I20240503 14:12:25.457789 28030 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:25.459656 28600 tablet_bootstrap.cc:492] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.459837 28600 ts_tablet_manager.cc:1367] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
I20240503 14:12:25.460927 28600 raft_consensus.cc:348] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.461243 28600 raft_consensus.cc:374] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.461354 28600 raft_consensus.cc:729] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.461623 28600 consensus_queue.cc:260] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.462353 28600 ts_tablet_manager.cc:1398] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:25.462862 28600 tablet_bootstrap.cc:492] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.464586 28600 tablet_bootstrap.cc:654] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.478644 28614 tablet_bootstrap.cc:492] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:25.478864 28614 ts_tablet_manager.cc:1367] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.139s user 0.009s sys 0.000s
I20240503 14:12:25.479696 28614 raft_consensus.cc:348] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.479966 28614 raft_consensus.cc:374] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.480026 28614 raft_consensus.cc:729] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.480270 28614 consensus_queue.cc:260] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.480963 28614 ts_tablet_manager.cc:1398] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:25.481312 28614 tablet_bootstrap.cc:492] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:25.485095 28661 ts_tablet_manager.cc:1888] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.488109 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7e1f3443c298426db9ff5df1437884f9
I20240503 14:12:25.488754 28031 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1103) took 1486 ms. Trace:
I20240503 14:12:25.488858 28031 rpcz_store.cc:276] 0503 14:12:24.002547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.503885 (+501338us) service_pool.cc:225] Handling call
0503 14:12:24.504393 (+ 508us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:24.504511 (+ 118us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:24.504579 (+ 68us) catalog_manager.cc:2758] Locking tablets
0503 14:12:24.530097 (+ 25518us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:25.487838 (+957741us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:25.487961 (+ 123us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:25.488020 (+ 59us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:25.488730 (+ 710us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:24.552171 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.552262 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.552267 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:24.552269 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:24.552357 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.552366 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.552371 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:24.552463 (+ 92us) tablet.cc:699] Row locks acquired
0503 14:12:24.552465 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:24.552494 (+ 29us) write_op.cc:270] Start()
0503 14:12:24.552517 (+ 23us) write_op.cc:276] Timestamp: P: 1714745544552473 usec, L: 0
0503 14:12:24.552521 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.552653 (+ 132us) log.cc:842] Serialized 762 byte log entry
0503 14:12:25.487050 (+934397us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.487226 (+ 176us) write_op.cc:301] APPLY: starting
0503 14:12:25.487243 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.487290 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.487318 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.487404 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.487442 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:25.487450 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.487538 (+ 88us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.487617 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.487638 (+ 21us) write_op.cc:454] Released schema lock
0503 14:12:25.487646 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.487703 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.487707 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":116,"apply.run_cpu_time_us":536,"apply.run_wall_time_us":536,"num_ops":2,"prepare.queue_time_us":21315,"prepare.run_cpu_time_us":544,"prepare.run_wall_time_us":811,"replication_time_us":934499}]]}
I20240503 14:12:25.489318 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1f3443c298426db9ff5df1437884f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:24 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.492484 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b0993be031cc4792a9eadba9ce41d024
I20240503 14:12:25.492533 28661 log.cc:1196] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/903f11a608b1483887d9b5234a404470
I20240503 14:12:25.431927 28604 tablet_bootstrap.cc:492] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.493053 28028 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1105) took 1402 ms. Trace:
I20240503 14:12:25.493345 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet b0993be031cc4792a9eadba9ce41d024 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:24 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.493145 28028 rpcz_store.cc:276] 0503 14:12:24.090695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.528864 (+438169us) service_pool.cc:225] Handling call
0503 14:12:24.546096 (+ 17232us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedfca6 00007fe123ed1f66 00007fe123ed1c95 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:24.546501 (+ 405us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:24.546619 (+ 118us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:24.546681 (+ 62us) catalog_manager.cc:2758] Locking tablets
0503 14:12:24.546813 (+ 132us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:25.492328 (+945515us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:25.492409 (+ 81us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:25.492453 (+ 44us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:25.493033 (+ 580us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:24.553764 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.553860 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.553865 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:24.553867 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:24.553948 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.553956 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.553961 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:24.554037 (+ 76us) tablet.cc:699] Row locks acquired
0503 14:12:24.554040 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:24.554065 (+ 25us) write_op.cc:270] Start()
0503 14:12:24.554087 (+ 22us) write_op.cc:276] Timestamp: P: 1714745544554045 usec, L: 0
0503 14:12:24.554091 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.554241 (+ 150us) log.cc:842] Serialized 762 byte log entry
0503 14:12:25.488279 (+934038us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.488347 (+ 68us) write_op.cc:301] APPLY: starting
0503 14:12:25.488358 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.488388 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.488410 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.488458 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.488484 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:25.488491 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.488543 (+ 52us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.488590 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.488606 (+ 16us) write_op.cc:454] Released schema lock
0503 14:12:25.488611 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.488664 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.488667 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":39055104,"child_traces":[["op",{"apply.queue_time_us":32,"apply.run_cpu_time_us":361,"apply.run_wall_time_us":362,"num_ops":2,"prepare.queue_time_us":6296,"prepare.run_cpu_time_us":538,"prepare.run_wall_time_us":789,"replication_time_us":934170}]]}
I20240503 14:12:25.493906 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a635bdd722254d13bf665ab69c76c554" } new_table_name: "default.Table_afb71497f321446fa44bc40c042d1db4" modify_external_catalogs: true
I20240503 14:12:25.494655 28604 tablet_bootstrap.cc:654] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:25.495474 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 218481d1ddd246cf9aa9c0e163666f4e consensus state has no leader
I20240503 14:12:25.495630 28027 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1104) took 1410 ms. Trace:
I20240503 14:12:25.495699 28027 rpcz_store.cc:276] 0503 14:12:24.085129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.507842 (+422713us) service_pool.cc:225] Handling call
0503 14:12:24.544589 (+ 36747us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cee0f46 00007fe123ed3c26 00007fe123ed3955 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:24.544891 (+ 302us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:24.545111 (+ 220us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:24.545332 (+ 221us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:24.545398 (+ 66us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:25.492047 (+946649us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:25.495613 (+ 3566us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:24.553013 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.553094 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.553099 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:24.553102 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:24.553182 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.553191 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.553195 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:24.553248 (+ 53us) tablet.cc:699] Row locks acquired
0503 14:12:24.553251 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:24.553276 (+ 25us) write_op.cc:270] Start()
0503 14:12:24.553298 (+ 22us) write_op.cc:276] Timestamp: P: 1714745544553256 usec, L: 0
0503 14:12:24.553301 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.553416 (+ 115us) log.cc:842] Serialized 405 byte log entry
0503 14:12:25.487144 (+933728us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.487802 (+ 658us) write_op.cc:301] APPLY: starting
0503 14:12:25.487814 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.487843 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.487865 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.487910 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.487936 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.487943 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.488005 (+ 62us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.488054 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.488074 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:25.488098 (+ 24us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.488148 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.488152 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":83845376,"child_traces":[["op",{"apply.queue_time_us":624,"apply.run_cpu_time_us":405,"apply.run_wall_time_us":404,"num_ops":1,"prepare.queue_time_us":7094,"prepare.run_cpu_time_us":462,"prepare.run_wall_time_us":720,"replication_time_us":933828}]]}
I20240503 14:12:25.496551 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8220e8ba6ccb4f89ba7481c40f8af3ed" } modify_external_catalogs: true
I20240503 14:12:25.496721 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8220e8ba6ccb4f89ba7481c40f8af3ed" } modify_external_catalogs: true
I20240503 14:12:25.499110 28661 ts_tablet_manager.cc:1909] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:25.499702 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 850) took 4244 ms. Trace:
I20240503 14:12:25.500185 28600 tablet_bootstrap.cc:492] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.502141 28600 ts_tablet_manager.cc:1367] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.001s
W20240503 14:12:25.506080 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7e1f3443c298426db9ff5df1437884f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1f3443c298426db9ff5df1437884f9 already in progress: creating tablet
I20240503 14:12:25.500703 29811 consensus_queue.cc:1035] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:25.506430 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b0993be031cc4792a9eadba9ce41d024 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0993be031cc4792a9eadba9ce41d024 already in progress: creating tablet
I20240503 14:12:25.504545 30129 raft_consensus.cc:482] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.507122 28600 raft_consensus.cc:348] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.507349 28600 raft_consensus.cc:374] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.507422 28600 raft_consensus.cc:729] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.507452 30129 raft_consensus.cc:504] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.507648 28600 consensus_queue.cc:260] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.508339 30129 leader_election.cc:288] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.516516 28600 ts_tablet_manager.cc:1398] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:12:25.516947 28600 tablet_bootstrap.cc:492] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.500824 28150 leader_election.cc:302] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:25.520061 30129 raft_consensus.cc:2781] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.520251 30129 raft_consensus.cc:482] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.520385 30129 raft_consensus.cc:3037] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.526618 30129 raft_consensus.cc:504] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.527457 30129 leader_election.cc:288] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:25.502067 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 66110.8us
I20240503 14:12:25.527788 28661 rpcz_store.cc:276] 0503 14:12:21.255086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.255469 (+ 383us) service_pool.cc:225] Handling call
0503 14:12:25.437945 (+4182476us) ts_tablet_manager.cc:1120] Deleting tablet 903f11a608b1483887d9b5234a404470
0503 14:12:25.437957 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.450069 (+ 12112us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.484949 (+ 34880us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.499663 (+ 14714us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4182238}
I20240503 14:12:25.528384 28661 tablet_replica.cc:330] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:25.528667 28661 raft_consensus.cc:2229] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:25.529053 29750 raft_consensus.cc:482] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.529207 29750 raft_consensus.cc:504] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.529790 29750 leader_election.cc:288] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.529884 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet bf21f6234dd64cf8a32f1599fe50746e (table default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]) successfully deleted
I20240503 14:12:25.530500 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd405fd84d44076b1792b7fcbe00e84" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.530923 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:25.531122 28212 raft_consensus.cc:2379] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:25.532903 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet b0993be031cc4792a9eadba9ce41d024 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:24 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.534641 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1f3443c298426db9ff5df1437884f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:24 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.537182 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93fea5caec94695a5df26d3630c50c7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.537614 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:25.537850 28488 raft_consensus.cc:2379] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:25.539772 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 903f11a608b1483887d9b5234a404470 (table default.Table_4afe0b312f964d3885779a752ec8c256 [id=32be054076954aed9265888ca62e510b]) successfully deleted
I20240503 14:12:25.541155 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab077988364bd687e54f90c3c265d5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.541337 28214 raft_consensus.cc:2454] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:25.542220 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6720048b4eb4201a75c15cf8031069f consensus state has no leader
I20240503 14:12:25.542284 29750 consensus_queue.cc:1035] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.542438 28028 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1106) took 1449 ms. Trace:
I20240503 14:12:25.542538 28028 rpcz_store.cc:276] 0503 14:12:24.092687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.493755 (+1401068us) service_pool.cc:225] Handling call
0503 14:12:25.494100 (+ 345us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:25.494343 (+ 243us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:25.494539 (+ 196us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:25.494623 (+ 84us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:25.541957 (+ 47334us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:25.542382 (+ 425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.517544 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.517657 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.517664 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.517675 (+ 11us) tablet.cc:660] Decoding operations
0503 14:12:25.517777 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.517790 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.517795 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:25.517868 (+ 73us) tablet.cc:699] Row locks acquired
0503 14:12:25.517870 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.517913 (+ 43us) write_op.cc:270] Start()
0503 14:12:25.517946 (+ 33us) write_op.cc:276] Timestamp: P: 1714745545517900 usec, L: 0
0503 14:12:25.517950 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.518144 (+ 194us) log.cc:842] Serialized 405 byte log entry
0503 14:12:25.539311 (+ 21167us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.539601 (+ 290us) write_op.cc:301] APPLY: starting
0503 14:12:25.539618 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.539666 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.539695 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.539752 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.539793 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.539801 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.539893 (+ 92us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.539964 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.539986 (+ 22us) write_op.cc:454] Released schema lock
0503 14:12:25.539994 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.540052 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.540056 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":232,"apply.run_cpu_time_us":536,"apply.run_wall_time_us":1041,"num_ops":1,"prepare.queue_time_us":22428,"prepare.run_cpu_time_us":717,"prepare.run_wall_time_us":1095,"raft.queue_time_us":33,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":21329}]]}
I20240503 14:12:25.543009 28427 leader_election.cc:302] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.543313 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cfcbfe39af47450d9a0917259530eb81" } modify_external_catalogs: true
I20240503 14:12:25.543543 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cfcbfe39af47450d9a0917259530eb81" } modify_external_catalogs: true
I20240503 14:12:25.543645 29994 raft_consensus.cc:2781] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.543720 29994 raft_consensus.cc:482] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.543772 29994 raft_consensus.cc:3037] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.544422 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5087cf4a795f4ee08ce4dd19d095d0e5
I20240503 14:12:25.544929 28027 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1107) took 1427 ms. Trace:
I20240503 14:12:25.545014 28027 rpcz_store.cc:276] 0503 14:12:24.117479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.496372 (+1378893us) service_pool.cc:225] Handling call
0503 14:12:25.496823 (+ 451us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:25.496959 (+ 136us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:25.497027 (+ 68us) catalog_manager.cc:2758] Locking tablets
0503 14:12:25.497158 (+ 131us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:25.544262 (+ 47104us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:25.544352 (+ 90us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:25.544396 (+ 44us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:25.544878 (+ 482us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.518664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.518778 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.518784 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.518786 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.518894 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.518905 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.518910 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.519002 (+ 92us) tablet.cc:699] Row locks acquired
0503 14:12:25.519005 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.519036 (+ 31us) write_op.cc:270] Start()
0503 14:12:25.519061 (+ 25us) write_op.cc:276] Timestamp: P: 1714745545519025 usec, L: 0
0503 14:12:25.519065 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.519238 (+ 173us) log.cc:842] Serialized 731 byte log entry
0503 14:12:25.539385 (+ 20147us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.540159 (+ 774us) write_op.cc:301] APPLY: starting
0503 14:12:25.540179 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.540216 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.540243 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.540305 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.540336 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:25.540343 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.540408 (+ 65us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.540452 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.540471 (+ 19us) write_op.cc:454] Released schema lock
0503 14:12:25.540477 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.540533 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.540537 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":727,"apply.run_cpu_time_us":430,"apply.run_wall_time_us":430,"mutex_wait_us":3,"num_ops":2,"prepare.queue_time_us":20809,"prepare.run_cpu_time_us":935,"prepare.run_wall_time_us":20098,"raft.queue_time_us":422,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":20306,"spinlock_wait_cycles":10624,"thread_start_us":19061,"threads_started":1}]]}
I20240503 14:12:25.545656 28614 tablet_bootstrap.cc:654] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.545727 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet e4a544fc27624dcc8a4cb003e59408f2 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:25.546059 29811 raft_consensus.cc:482] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.546208 29811 raft_consensus.cc:504] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.546321 29994 raft_consensus.cc:504] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.546855 29811 leader_election.cc:288] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.546926 29994 leader_election.cc:288] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:25.483525 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 203152us
I20240503 14:12:25.547801 30130 consensus_queue.cc:1035] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.548298 30070 consensus_queue.cc:1035] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.538846 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93fea5caec94695a5df26d3630c50c7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.548907 28357 raft_consensus.cc:2454] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.539054 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.529067 28661 raft_consensus.cc:2258] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.555291 29828 ts_tablet_manager.cc:1888] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:25.555436 29828 log.cc:1196] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/c5297664a4d943c28adfb6ac4f061680
I20240503 14:12:25.555774 29828 ts_tablet_manager.cc:1909] T c5297664a4d943c28adfb6ac4f061680 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:25.556926 29828 tablet_replica.cc:330] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:25.557163 29828 raft_consensus.cc:2229] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:25.558849 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.559072 28600 tablet_bootstrap.cc:654] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.561189 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69425d29d55b4fe4bb7f08ae390aaa86" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.561698 28604 tablet_bootstrap.cc:492] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.561714 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.561853 28604 ts_tablet_manager.cc:1367] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.130s user 0.002s sys 0.002s
I20240503 14:12:25.562347 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e4a544fc27624dcc8a4cb003e59408f2
I20240503 14:12:25.562387 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab077988364bd687e54f90c3c265d5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:25.562597 28215 raft_consensus.cc:3037] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.562659 28604 raft_consensus.cc:348] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.562834 28604 raft_consensus.cc:374] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.562865 28027 catalog_manager.cc:6554] ReplaceTablet: tablet e4a544fc27624dcc8a4cb003e59408f2 deleted and replaced by tablet 773bc744579b4ff28c950ea632753221
I20240503 14:12:25.562881 28604 raft_consensus.cc:729] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.562987 28027 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1109) took 1432 ms. Trace:
I20240503 14:12:25.563114 28604 consensus_queue.cc:260] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.563087 28027 rpcz_store.cc:276] 0503 14:12:24.130765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.545624 (+1414859us) service_pool.cc:225] Handling call
0503 14:12:25.562957 (+ 17333us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.555456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.555593 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.555600 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.555602 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.555749 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.555765 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.555770 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.555856 (+ 86us) tablet.cc:699] Row locks acquired
0503 14:12:25.555858 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.555910 (+ 52us) write_op.cc:270] Start()
0503 14:12:25.555953 (+ 43us) write_op.cc:276] Timestamp: P: 1714745545555891 usec, L: 0
0503 14:12:25.555957 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.556232 (+ 275us) log.cc:842] Serialized 628 byte log entry
0503 14:12:25.560358 (+ 4126us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.560513 (+ 155us) write_op.cc:301] APPLY: starting
0503 14:12:25.560532 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.560583 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.560611 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.560746 (+ 135us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.560812 (+ 66us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.560821 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:25.560917 (+ 96us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.560989 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.561015 (+ 26us) write_op.cc:454] Released schema lock
0503 14:12:25.561023 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.561089 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.561092 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":638,"apply.run_wall_time_us":638,"mutex_wait_us":388,"num_ops":2,"prepare.queue_time_us":8615,"prepare.run_cpu_time_us":968,"prepare.run_wall_time_us":1871,"raft.queue_time_us":465,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":300,"replication_time_us":4371}]]}
I20240503 14:12:25.563616 28604 ts_tablet_manager.cc:1398] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:25.564538 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7c688a2e434147e582bcf88eeafc227b" } modify_external_catalogs: true
I20240503 14:12:25.563905 28604 tablet_bootstrap.cc:492] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.564961 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7c688a2e434147e582bcf88eeafc227b" } modify_external_catalogs: true
I20240503 14:12:25.565320 28215 raft_consensus.cc:2454] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:25.571110 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a544fc27624dcc8a4cb003e59408f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 773bc744579b4ff28c950ea632753221) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.572741 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69425d29d55b4fe4bb7f08ae390aaa86" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.573807 28150 leader_election.cc:302] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
W20240503 14:12:25.574173 28150 leader_election.cc:341] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:25.574457 28150 leader_election.cc:341] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.574618 29811 raft_consensus.cc:2781] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.574720 29811 raft_consensus.cc:482] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.574779 29811 raft_consensus.cc:3037] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:25.574756 28150 leader_election.cc:341] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.575040 28150 leader_election.cc:302] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:25.575564 28150 leader_election.cc:341] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.575604 30005 raft_consensus.cc:2726] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.575649 28150 leader_election.cc:302] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.575968 30005 raft_consensus.cc:2726] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.583467 28427 leader_election.cc:302] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:25.584123 28427 leader_election.cc:302] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:25.584146 29750 raft_consensus.cc:2726] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.584570 29750 raft_consensus.cc:2726] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.587136 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137ef4cfc2476cbc70f1cc72ad89e6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:25.587365 28488 raft_consensus.cc:3037] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.589747 28604 tablet_bootstrap.cc:654] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.589980 28488 raft_consensus.cc:2454] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:25.591449 29811 raft_consensus.cc:504] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.592124 29811 leader_election.cc:288] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.592816 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93fea5caec94695a5df26d3630c50c7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:25.592876 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c77a11ef384709aef5b62273380573" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.593057 28487 raft_consensus.cc:2454] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:25.593768 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c77a11ef384709aef5b62273380573" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.593977 28212 raft_consensus.cc:2454] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:25.594525 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1958dc488e4854a8c8e74da9fab243" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.595479 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137ef4cfc2476cbc70f1cc72ad89e6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:25.595546 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1f3443c298426db9ff5df1437884f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:24 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.595711 28214 raft_consensus.cc:3037] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.595927 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet b0993be031cc4792a9eadba9ce41d024 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:24 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.596241 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93fea5caec94695a5df26d3630c50c7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.596299 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.596441 28352 raft_consensus.cc:3037] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.596580 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a544fc27624dcc8a4cb003e59408f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 773bc744579b4ff28c950ea632753221) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.597041 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet bf21f6234dd64cf8a32f1599fe50746e (table default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]) successfully deleted
I20240503 14:12:25.597396 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1958dc488e4854a8c8e74da9fab243" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.597664 28488 raft_consensus.cc:2379] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:25.598839 28214 raft_consensus.cc:2454] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:25.599099 28352 raft_consensus.cc:2454] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:25.600816 28427 leader_election.cc:302] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:25.601158 29750 raft_consensus.cc:2726] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.602901 28150 leader_election.cc:302] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.604606 28614 tablet_bootstrap.cc:492] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:25.604763 28614 ts_tablet_manager.cc:1367] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.124s user 0.001s sys 0.002s
I20240503 14:12:25.603888 28600 tablet_bootstrap.cc:492] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.605477 28600 ts_tablet_manager.cc:1367] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.000s
I20240503 14:12:25.606316 28600 raft_consensus.cc:348] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.606595 28600 raft_consensus.cc:374] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.606700 28600 raft_consensus.cc:729] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.607352 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2946ba624b244db9a617f2e202bfc1ec
I20240503 14:12:25.607858 28028 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1108) took 1485 ms. Trace:
I20240503 14:12:25.608269 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 2946ba624b244db9a617f2e202bfc1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.607961 28028 rpcz_store.cc:276] 0503 14:12:24.122586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.543135 (+1420549us) service_pool.cc:225] Handling call
0503 14:12:25.543657 (+ 522us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:25.543816 (+ 159us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:25.543896 (+ 80us) catalog_manager.cc:2758] Locking tablets
0503 14:12:25.593459 (+ 49563us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:25.607131 (+ 13672us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:25.607223 (+ 92us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:25.607293 (+ 70us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:25.607798 (+ 505us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.594302 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.594412 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.594418 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.594420 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.594535 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.594549 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.594554 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.594648 (+ 94us) tablet.cc:699] Row locks acquired
0503 14:12:25.594651 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.594705 (+ 54us) write_op.cc:270] Start()
0503 14:12:25.594740 (+ 35us) write_op.cc:276] Timestamp: P: 1714745545594685 usec, L: 0
0503 14:12:25.594743 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.594921 (+ 178us) log.cc:842] Serialized 762 byte log entry
0503 14:12:25.606464 (+ 11543us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.606602 (+ 138us) write_op.cc:301] APPLY: starting
0503 14:12:25.606618 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.606670 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.606710 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.606804 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.606845 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:25.606854 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:25.606938 (+ 84us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.607011 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.607033 (+ 22us) write_op.cc:454] Released schema lock
0503 14:12:25.607041 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.607105 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.607109 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":566,"apply.run_wall_time_us":565,"mutex_wait_us":317,"num_ops":2,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":758,"prepare.run_wall_time_us":1065,"raft.queue_time_us":400,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":246,"replication_time_us":11696}]]}
I20240503 14:12:25.608567 28614 raft_consensus.cc:348] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.608769 28614 raft_consensus.cc:374] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.608824 28614 raft_consensus.cc:729] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.608892 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8d3a69435d7b492c88dd63b58b055c6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:25.609011 28614 consensus_queue.cc:260] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.609298 30142 tablet_replica.cc:330] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:25.609467 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d3a69435d7b492c88dd63b58b055c6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.609493 30142 raft_consensus.cc:2229] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:25.609545 28614 ts_tablet_manager.cc:1398] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 14:12:25.609688 30142 raft_consensus.cc:2258] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.609841 28614 tablet_bootstrap.cc:492] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:25.610252 30142 ts_tablet_manager.cc:1875] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.611212 28614 tablet_bootstrap.cc:654] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:25.613240 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5087cf4a795f4ee08ce4dd19d095d0e5 already in progress: creating tablet
I20240503 14:12:25.613561 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7b89711872ba442ca66e35091a2271da
I20240503 14:12:25.613812 28427 leader_election.cc:302] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.614042 28027 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1110) took 1439 ms. Trace:
I20240503 14:12:25.614095 29994 raft_consensus.cc:2781] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.614349 29994 raft_consensus.cc:686] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:25.614151 28027 rpcz_store.cc:276] 0503 14:12:24.174928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.564359 (+1389431us) service_pool.cc:225] Handling call
0503 14:12:25.565141 (+ 782us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:25.565288 (+ 147us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:25.565359 (+ 71us) catalog_manager.cc:2758] Locking tablets
0503 14:12:25.604235 (+ 38876us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:25.613377 (+ 9142us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:25.613465 (+ 88us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:25.613509 (+ 44us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:25.613992 (+ 483us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.605219 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.605354 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.605361 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.605364 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.605493 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.605505 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.605510 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.605623 (+ 113us) tablet.cc:699] Row locks acquired
0503 14:12:25.605625 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.605669 (+ 44us) write_op.cc:270] Start()
0503 14:12:25.605708 (+ 39us) write_op.cc:276] Timestamp: P: 1714745545605649 usec, L: 0
0503 14:12:25.605712 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.605906 (+ 194us) log.cc:842] Serialized 731 byte log entry
0503 14:12:25.612505 (+ 6599us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.612651 (+ 146us) write_op.cc:301] APPLY: starting
0503 14:12:25.612668 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.612722 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.612750 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.612819 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.612853 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:25.612860 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.612932 (+ 72us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.613002 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.613019 (+ 17us) write_op.cc:454] Released schema lock
0503 14:12:25.613025 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.613080 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.613084 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":484,"apply.run_wall_time_us":485,"num_ops":2,"prepare.queue_time_us":132,"prepare.run_cpu_time_us":803,"prepare.run_wall_time_us":804,"raft.queue_time_us":73,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":6769}]]}
I20240503 14:12:25.614605 29994 consensus_queue.cc:237] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.614971 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4237d991665c4015b7d01b31d99f2370"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:25.615485 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4237d991665c4015b7d01b31d99f2370 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.617414 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b89711872ba442ca66e35091a2271da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.620308 29828 raft_consensus.cc:2258] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.620281 28028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1111) took 1242 ms. Trace:
I20240503 14:12:25.620882 28028 rpcz_store.cc:276] 0503 14:12:24.377875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.608610 (+1230735us) service_pool.cc:225] Handling call
0503 14:12:25.609567 (+ 957us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.610068 (+ 501us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.619207 (+ 9139us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.620077 (+ 870us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.620208 (+ 131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.611714 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.611837 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.611844 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.611846 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.611941 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.611953 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.611958 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.612038 (+ 80us) tablet.cc:699] Row locks acquired
0503 14:12:25.612041 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.612077 (+ 36us) write_op.cc:270] Start()
0503 14:12:25.612146 (+ 69us) write_op.cc:276] Timestamp: P: 1714745545612058 usec, L: 0
0503 14:12:25.612151 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.612334 (+ 183us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.617545 (+ 5211us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.617710 (+ 165us) write_op.cc:301] APPLY: starting
0503 14:12:25.617726 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.617771 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.617796 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.617907 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.617942 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.617949 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.618023 (+ 74us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.618088 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.618106 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:25.618112 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.618164 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.618167 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":111,"apply.run_cpu_time_us":505,"apply.run_wall_time_us":505,"mutex_wait_us":294,"num_ops":2,"prepare.queue_time_us":1159,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":1050,"raft.queue_time_us":360,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":5370,"spinlock_wait_cycles":22784}]]}
I20240503 14:12:25.622540 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0573272ac5c84afc82be96eea79f4ce3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:25.623260 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0573272ac5c84afc82be96eea79f4ce3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.606963 28600 consensus_queue.cc:260] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.626780 28600 ts_tablet_manager.cc:1398] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240503 14:12:25.627560 28027 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1112) took 1249 ms. Trace:
I20240503 14:12:25.627728 28027 rpcz_store.cc:276] 0503 14:12:24.377984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.614736 (+1236752us) service_pool.cc:225] Handling call
0503 14:12:25.615576 (+ 840us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.616022 (+ 446us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.627252 (+ 11230us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.627378 (+ 126us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.627478 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.617842 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.617960 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.617966 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.617969 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.618066 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.618078 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.618083 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.618154 (+ 71us) tablet.cc:699] Row locks acquired
0503 14:12:25.618156 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.618186 (+ 30us) write_op.cc:270] Start()
0503 14:12:25.618213 (+ 27us) write_op.cc:276] Timestamp: P: 1714745545618168 usec, L: 0
0503 14:12:25.618217 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.618362 (+ 145us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.626483 (+ 8121us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.626615 (+ 132us) write_op.cc:301] APPLY: starting
0503 14:12:25.626633 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.626690 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.626719 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.626862 (+ 143us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.626899 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.626907 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.626986 (+ 79us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.627057 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.627077 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:25.627085 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.627139 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.627143 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":578,"apply.run_wall_time_us":577,"num_ops":2,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":1104,"raft.queue_time_us":61,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":8240}]]}
I20240503 14:12:25.629921 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 6a27d21de6a6441d810f379bba88f9a2 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:25.631183 28614 tablet_bootstrap.cc:492] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:25.631353 28614 ts_tablet_manager.cc:1367] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20240503 14:12:25.632860 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a544fc27624dcc8a4cb003e59408f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 773bc744579b4ff28c950ea632753221) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.632966 29828 ts_tablet_manager.cc:1875] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.634092 28028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1113) took 1237 ms. Trace:
I20240503 14:12:25.634303 28028 rpcz_store.cc:276] 0503 14:12:24.396140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.622287 (+1226147us) service_pool.cc:225] Handling call
0503 14:12:25.623419 (+ 1132us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.623944 (+ 525us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.633385 (+ 9441us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.633525 (+ 140us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.633656 (+ 131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.624546 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.624653 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.624660 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.624662 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.624767 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.624779 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.624784 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.624864 (+ 80us) tablet.cc:699] Row locks acquired
0503 14:12:25.624867 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.624902 (+ 35us) write_op.cc:270] Start()
0503 14:12:25.624936 (+ 34us) write_op.cc:276] Timestamp: P: 1714745545624883 usec, L: 0
0503 14:12:25.624940 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.625110 (+ 170us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.630194 (+ 5084us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.630357 (+ 163us) write_op.cc:301] APPLY: starting
0503 14:12:25.630373 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.630413 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.630439 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.630549 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.630582 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.630589 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.630662 (+ 73us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.630742 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.630761 (+ 19us) write_op.cc:454] Released schema lock
0503 14:12:25.630768 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.630822 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.630826 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":110,"apply.run_cpu_time_us":517,"apply.run_wall_time_us":517,"num_ops":2,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":667,"prepare.run_wall_time_us":667,"raft.queue_time_us":61,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":5227}]]}
I20240503 14:12:25.635991 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4f391faa840c44368eba424683091351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:25.636413 28614 raft_consensus.cc:348] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.636617 28614 raft_consensus.cc:374] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.636679 28614 raft_consensus.cc:729] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.636718 28600 tablet_bootstrap.cc:492] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.636889 28614 consensus_queue.cc:260] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:25.637888 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f391faa840c44368eba424683091351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.638841 28614 ts_tablet_manager.cc:1398] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:25.639164 28614 tablet_bootstrap.cc:492] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:25.638895 28150 leader_election.cc:341] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.640504 28614 tablet_bootstrap.cc:654] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.639122 30005 raft_consensus.cc:2781] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.641110 30005 raft_consensus.cc:686] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:25.641384 30005 consensus_queue.cc:237] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.644873 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet e94673f00cb3432185009bda2ce398fa (table default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d [id=2849f6ba9ef64d26a5dc65f9ac9048da]). Expected version 1 got 0
I20240503 14:12:25.645387 28032 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:25.648305 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.648556 28347 raft_consensus.cc:2454] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:25.649212 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 2946ba624b244db9a617f2e202bfc1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.649686 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b89711872ba442ca66e35091a2271da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.650174 28604 tablet_bootstrap.cc:492] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.650329 28604 ts_tablet_manager.cc:1367] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.001s
I20240503 14:12:25.650722 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:25.651031 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet e94673f00cb3432185009bda2ce398fa (table default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d [id=2849f6ba9ef64d26a5dc65f9ac9048da]). Expected version 1 got 0
I20240503 14:12:25.651302 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet b75036cdf9e04ea4aa767fd12016f177 (table default.Table_87bcc8d1e5244ebdb6ed58c62b34f709 [id=30ff854aabe84b2cb03824d287ae5685]). Expected version 1 got 0
I20240503 14:12:25.651535 28604 raft_consensus.cc:348] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.651726 28604 raft_consensus.cc:374] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.651779 28604 raft_consensus.cc:729] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.651960 28604 consensus_queue.cc:260] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.652215 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_6595add4a286443f9cfe220b536c5709 [id=12558f7a877545f0900925e9c4948420] alter complete (version 2)
I20240503 14:12:25.652658 28604 ts_tablet_manager.cc:1398] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:25.652940 28604 tablet_bootstrap.cc:492] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:25.653635 28033 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.193:45997 (request call id 95) took 2593 ms (client timeout 1000 ms). Trace:
I20240503 14:12:25.654017 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.654577 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6202333af943cbbc0d09553c1c193c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:25.653787 28033 rpcz_store.cc:269] 0503 14:12:23.060514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.331858 (+271344us) service_pool.cc:225] Handling call
0503 14:12:23.356524 (+ 24666us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:25.653587 (+2297063us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:24.267422 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.267518 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.267524 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:24.267526 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:24.267617 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.267626 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.267630 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:24.267701 (+ 71us) tablet.cc:699] Row locks acquired
0503 14:12:24.267704 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:24.267737 (+ 33us) write_op.cc:270] Start()
0503 14:12:24.267773 (+ 36us) write_op.cc:276] Timestamp: P: 1714745544267700 usec, L: 0
0503 14:12:24.267776 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.267914 (+ 138us) log.cc:842] Serialized 533 byte log entry
0503 14:12:24.350237 (+ 82323us) op_driver.cc:444] REPLICATION: finished
0503 14:12:24.350323 (+ 86us) write_op.cc:301] APPLY: starting
0503 14:12:24.350339 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:24.350398 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:24.350424 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:24.350483 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:24.350519 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:24.350528 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:24.350641 (+ 113us) log.cc:842] Serialized 23 byte log entry
0503 14:12:24.350713 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:24.350733 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:24.350742 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:24.350831 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:24.350834 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0503 14:12:24.371360 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.371516 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.371525 (+ 9us) write_op.cc:435] Acquired schema lock
0503 14:12:24.371528 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:24.371667 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.371686 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.371692 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:24.371774 (+ 82us) tablet.cc:699] Row locks acquired
0503 14:12:24.371777 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:24.371826 (+ 49us) write_op.cc:270] Start()
0503 14:12:24.371869 (+ 43us) write_op.cc:276] Timestamp: P: 1714745544371776 usec, L: 0
0503 14:12:24.371873 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.372139 (+ 266us) log.cc:842] Serialized 356 byte log entry
0503 14:12:24.475068 (+102929us) op_driver.cc:444] REPLICATION: finished
0503 14:12:24.475484 (+ 416us) write_op.cc:301] APPLY: starting
0503 14:12:24.475505 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:24.475558 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:24.475590 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:24.475662 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:24.475703 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:24.475711 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:24.475814 (+ 103us) log.cc:842] Serialized 23 byte log entry
0503 14:12:24.475899 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:24.475923 (+ 24us) write_op.cc:454] Released schema lock
0503 14:12:24.475932 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:24.476001 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:24.476005 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0503 14:12:24.550173 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.550323 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.550330 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:24.550333 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:24.550527 (+ 194us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.550538 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.550543 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:24.550607 (+ 64us) tablet.cc:699] Row locks acquired
0503 14:12:24.550609 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:24.550645 (+ 36us) write_op.cc:270] Start()
0503 14:12:24.550673 (+ 28us) write_op.cc:276] Timestamp: P: 1714745544550622 usec, L: 0
0503 14:12:24.550677 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.550886 (+ 209us) log.cc:842] Serialized 356 byte log entry
0503 14:12:25.448462 (+897576us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.448565 (+ 103us) write_op.cc:301] APPLY: starting
0503 14:12:25.448585 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.448633 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.448677 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.448749 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.448795 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.448805 (+ 10us) write_op.cc:312] APPLY: finished
0503 14:12:25.448914 (+ 109us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.449002 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.449029 (+ 27us) write_op.cc:454] Released schema lock
0503 14:12:25.449039 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.449129 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.449133 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"spinlock_wait_cycles":54417792,"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":562,"num_ops":1,"prepare.queue_time_us":290343,"prepare.run_cpu_time_us":557,"prepare.run_wall_time_us":855,"replication_time_us":82426}],["op",{"apply.queue_time_us":350,"apply.run_cpu_time_us":580,"apply.run_wall_time_us":579,"num_ops":1,"prepare.queue_time_us":19194,"prepare.run_cpu_time_us":877,"prepare.run_wall_time_us":1309,"replication_time_us":103161}],["op",{"apply.queue_time_us":38,"apply.run_cpu_time_us":636,"apply.run_wall_time_us":636,"num_ops":1,"prepare.queue_time_us":25474,"prepare.run_cpu_time_us":705,"prepare.run_wall_time_us":1074,"replication_time_us":897752}]]}
I20240503 14:12:25.655025 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cd09e769be4bee95bbef9c3c550451 with delete_type TABLET_DATA_DELETED (replaced by tablet 49445f0f1dd1451c9ba066e61273f2f1) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.655181 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c8a07599c3614ce2ba13285c679960bb" } new_table_name: "default.Table_2e4155be94a44f6c8a8683824f24925a" modify_external_catalogs: true
I20240503 14:12:25.655472 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.655735 28604 tablet_bootstrap.cc:654] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.655928 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d28d3c2ea472191ad21355248a2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.656351 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221218847554cbd9d3242a9a6431e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.656754 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:25.658609 28024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 85) took 2600 ms (client timeout 1000 ms). Trace:
I20240503 14:12:25.659138 28661 ts_tablet_manager.cc:1875] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:25.658725 28024 rpcz_store.cc:269] 0503 14:12:23.057923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.071020 (+ 13097us) service_pool.cc:225] Handling call
0503 14:12:25.658565 (+2587545us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:23.960146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:23.960279 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:23.960287 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:23.960290 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:23.960407 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:23.960420 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:23.960426 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:23.960493 (+ 67us) tablet.cc:699] Row locks acquired
0503 14:12:23.960496 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:23.960537 (+ 41us) write_op.cc:270] Start()
0503 14:12:23.960574 (+ 37us) write_op.cc:276] Timestamp: P: 1714745543960522 usec, L: 0
0503 14:12:23.960577 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:23.960804 (+ 227us) log.cc:842] Serialized 533 byte log entry
0503 14:12:23.971634 (+ 10830us) op_driver.cc:444] REPLICATION: finished
0503 14:12:23.971943 (+ 309us) write_op.cc:301] APPLY: starting
0503 14:12:23.971962 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:23.972000 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:23.972055 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:23.972145 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:23.972188 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:23.972197 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:23.972276 (+ 79us) log.cc:842] Serialized 23 byte log entry
0503 14:12:23.972342 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.972362 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:23.972369 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:23.972434 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:23.972437 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0503 14:12:24.266481 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.266591 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.266597 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:24.266600 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:24.266693 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.266704 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.266709 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:24.266764 (+ 55us) tablet.cc:699] Row locks acquired
0503 14:12:24.266767 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:24.266800 (+ 33us) write_op.cc:270] Start()
0503 14:12:24.266831 (+ 31us) write_op.cc:276] Timestamp: P: 1714745544266763 usec, L: 0
0503 14:12:24.266835 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.266997 (+ 162us) log.cc:842] Serialized 356 byte log entry
0503 14:12:24.349098 (+ 82101us) op_driver.cc:444] REPLICATION: finished
0503 14:12:24.349216 (+ 118us) write_op.cc:301] APPLY: starting
0503 14:12:24.349249 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:24.349297 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:24.349359 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:24.349430 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:24.349474 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:24.349481 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:24.349586 (+ 105us) log.cc:842] Serialized 23 byte log entry
0503 14:12:24.349661 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:24.349692 (+ 31us) write_op.cc:454] Released schema lock
0503 14:12:24.349701 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:24.349773 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:24.349777 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":248,"apply.run_cpu_time_us":545,"apply.run_wall_time_us":546,"num_ops":1,"prepare.queue_time_us":93371,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":15896,"raft.queue_time_us":204,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"replication_time_us":11029}],["op",{"apply.queue_time_us":49,"apply.run_cpu_time_us":620,"apply.run_wall_time_us":620,"num_ops":1,"prepare.queue_time_us":292836,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":935,"replication_time_us":82233}]]}
I20240503 14:12:25.659843 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ac11e443561a400ea0c013dd4e9178cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:25.662496 28029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 86) took 1575 ms (client timeout 1000 ms). Trace:
W20240503 14:12:25.662597 28029 rpcz_store.cc:269] 0503 14:12:24.087215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.171852 (+ 84637us) service_pool.cc:225] Handling call
0503 14:12:25.662432 (+1490580us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":24}
I20240503 14:12:25.662829 28614 tablet_bootstrap.cc:492] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:25.663003 28614 ts_tablet_manager.cc:1367] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240503 14:12:25.663101 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f9bcc668aff94b09927c733f071bc6de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:25.663774 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9bcc668aff94b09927c733f071bc6de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.666280 28604 tablet_bootstrap.cc:492] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.666424 28604 ts_tablet_manager.cc:1367] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 14:12:25.667177 28604 raft_consensus.cc:348] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.667400 28604 raft_consensus.cc:374] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.667461 28604 raft_consensus.cc:729] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.667675 28604 consensus_queue.cc:260] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.673652 29996 raft_consensus.cc:482] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.673835 29996 raft_consensus.cc:504] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.674517 29996 leader_election.cc:288] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:25.675222 28030 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 106) took 879 ms (client timeout 1000 ms). Trace:
W20240503 14:12:25.675343 28030 rpcz_store.cc:269] 0503 14:12:24.795232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.207256 (+412024us) service_pool.cc:225] Handling call
0503 14:12:25.220489 (+ 13233us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:25.675137 (+454648us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.500793 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.500937 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.500945 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:25.500948 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.501080 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.501097 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.501103 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:25.501183 (+ 80us) tablet.cc:699] Row locks acquired
0503 14:12:25.501185 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.501230 (+ 45us) write_op.cc:270] Start()
0503 14:12:25.501270 (+ 40us) write_op.cc:276] Timestamp: P: 1714745545501217 usec, L: 0
0503 14:12:25.501274 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.501508 (+ 234us) log.cc:842] Serialized 533 byte log entry
0503 14:12:25.519927 (+ 18419us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.520126 (+ 199us) write_op.cc:301] APPLY: starting
0503 14:12:25.520142 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.520207 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.520236 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.520291 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.520331 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.520339 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.520432 (+ 93us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.520507 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.520524 (+ 17us) write_op.cc:454] Released schema lock
0503 14:12:25.520531 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.520591 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.520594 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6173,"reported_tablets":35,"spinlock_wait_cycles":27496832,"child_traces":[["op",{"apply.queue_time_us":131,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":522,"mutex_wait_us":1135,"num_ops":1,"prepare.queue_time_us":42149,"prepare.run_cpu_time_us":1020,"prepare.run_wall_time_us":16719,"raft.queue_time_us":1248,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":349,"replication_time_us":18629,"thread_start_us":43330,"threads_started":1}]]}
I20240503 14:12:25.676162 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 77f0081cc2cb48aaa368741109032fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676386 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221218847554cbd9d3242a9a6431e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676441 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a92fc3833e49aaab8c5415de9bb685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:07 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676386 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d28d3c2ea472191ad21355248a2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676630 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6202333af943cbbc0d09553c1c193c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676662 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f89fca0480d4bf28160384ae0421bce with delete_type TABLET_DATA_DELETED (replaced by tablet 62e4f6e3908e420e9fa156ee24b437d6) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676807 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676898 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af0342e0e0b47499a9322b5b2cb3d03 with delete_type TABLET_DATA_DELETED (replaced by tablet 70f0e07d32084999a3ff42f69bb1ca2f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676946 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cd09e769be4bee95bbef9c3c550451 with delete_type TABLET_DATA_DELETED (replaced by tablet 49445f0f1dd1451c9ba066e61273f2f1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.676810 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.677074 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet d43823da09bb49be8f5d1a1149a4b11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.677137 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.677417 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75036cdf9e04ea4aa767fd12016f177" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.677666 28215 raft_consensus.cc:2454] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:25.679636 28604 ts_tablet_manager.cc:1398] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
W20240503 14:12:25.679713 28025 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.193:45997 (request call id 96) took 1474 ms (client timeout 1000 ms). Trace:
I20240503 14:12:25.680312 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.680305 28427 leader_election.cc:302] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.680891 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fad4508a29d4822b81af6947b57e7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.680958 30134 raft_consensus.cc:2781] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.681031 30134 raft_consensus.cc:482] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.681092 30134 raft_consensus.cc:3037] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.681387 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbcae22ea124edaa14c9409f3052714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:25.681519 28026 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 105) took 1975 ms (client timeout 1000 ms). Trace:
W20240503 14:12:25.681636 28026 rpcz_store.cc:269] 0503 14:12:23.705840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.750082 (+ 44242us) service_pool.cc:225] Handling call
0503 14:12:23.780322 (+ 30240us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:25.681475 (+1901153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13393,"reported_tablets":26,"spinlock_wait_cycles":66967808}
I20240503 14:12:25.681831 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.682101 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e9e796aab9294a76b536ba18bd97568c" } new_table_name: "default.Table_bfecf31f8b1443d58376a8125ebcbff2" modify_external_catalogs: true
I20240503 14:12:25.682230 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.682735 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.683123 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet d43823da09bb49be8f5d1a1149a4b11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.683718 30134 raft_consensus.cc:504] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.684518 30134 leader_election.cc:288] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.684861 28604 tablet_bootstrap.cc:492] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.685552 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a77dd1752a4843739a371c414093b2c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:25.686503 28604 tablet_bootstrap.cc:654] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:25.686843 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a77dd1752a4843739a371c414093b2c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:25.688437 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac11e443561a400ea0c013dd4e9178cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.689191 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75036cdf9e04ea4aa767fd12016f177" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:25.689415 28215 raft_consensus.cc:3037] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:25.679872 28025 rpcz_store.cc:269] 0503 14:12:24.205393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.331352 (+125959us) service_pool.cc:225] Handling call
0503 14:12:24.368724 (+ 37372us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:25.679662 (+1310938us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:24.551289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:24.551385 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:24.551391 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:24.551393 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:24.551475 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:24.551484 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:24.551489 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:24.551617 (+ 128us) tablet.cc:699] Row locks acquired
0503 14:12:24.551619 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:24.551652 (+ 33us) write_op.cc:270] Start()
0503 14:12:24.551679 (+ 27us) write_op.cc:276] Timestamp: P: 1714745544551630 usec, L: 0
0503 14:12:24.551683 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:24.551813 (+ 130us) log.cc:842] Serialized 533 byte log entry
0503 14:12:25.448550 (+896737us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.449239 (+ 689us) write_op.cc:301] APPLY: starting
0503 14:12:25.449252 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.449295 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.449327 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.449377 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.449408 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.449416 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.449475 (+ 59us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.449542 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.449555 (+ 13us) write_op.cc:454] Released schema lock
0503 14:12:25.449561 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.449609 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.449613 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5515,"reported_tablets":21,"spinlock_wait_cycles":83422080,"child_traces":[["op",{"apply.queue_time_us":655,"apply.run_cpu_time_us":416,"apply.run_wall_time_us":415,"num_ops":1,"prepare.queue_time_us":26007,"prepare.run_cpu_time_us":547,"prepare.run_wall_time_us":851,"replication_time_us":896851,"thread_start_us":38380,"threads_started":1}]]}
I20240503 14:12:25.691723 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5d3fd3c6cb884647a2e2f0fe7cba999d" } new_table_name: "default.Table_e454f3eb809e4710a6ab2db46aa9644e" modify_external_catalogs: true
I20240503 14:12:25.692317 28215 raft_consensus.cc:2454] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:25.693485 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd405fd84d44076b1792b7fcbe00e84" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.693745 28350 raft_consensus.cc:2379] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:25.694227 28427 leader_election.cc:302] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.694963 29996 raft_consensus.cc:2781] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.695087 29996 raft_consensus.cc:686] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:25.695338 29996 consensus_queue.cc:237] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.697634 28032 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_0622233baec140adb2da5612db055913 [id=c2e9f50f96514774acb83f844cdd8692] alter complete (version 3)
I20240503 14:12:25.698590 30142 ts_tablet_manager.cc:1888] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.708389 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6a27d21de6a6441d810f379bba88f9a2
I20240503 14:12:25.710949 28027 catalog_manager.cc:6554] ReplaceTablet: tablet 6a27d21de6a6441d810f379bba88f9a2 deleted and replaced by tablet 0311cfbb060e439d9b7c497ffe4626cd
I20240503 14:12:25.711258 28027 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1114) took 1247 ms. Trace:
I20240503 14:12:25.711412 28614 raft_consensus.cc:348] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.711629 28614 raft_consensus.cc:374] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.711683 28614 raft_consensus.cc:729] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.711489 28027 rpcz_store.cc:276] 0503 14:12:24.463842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.629809 (+1165967us) service_pool.cc:225] Handling call
0503 14:12:25.711216 (+ 81407us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.632104 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.632216 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.632222 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.632224 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.632337 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.632351 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.632356 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.632440 (+ 84us) tablet.cc:699] Row locks acquired
0503 14:12:25.632443 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.632484 (+ 41us) write_op.cc:270] Start()
0503 14:12:25.632519 (+ 35us) write_op.cc:276] Timestamp: P: 1714745545632462 usec, L: 0
0503 14:12:25.632524 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.632721 (+ 197us) log.cc:842] Serialized 628 byte log entry
0503 14:12:25.651799 (+ 19078us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.651958 (+ 159us) write_op.cc:301] APPLY: starting
0503 14:12:25.651976 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.652029 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.652057 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.652234 (+ 177us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.652295 (+ 61us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.652303 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.652391 (+ 88us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.652456 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.652482 (+ 26us) write_op.cc:454] Released schema lock
0503 14:12:25.652491 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.652545 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.652549 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":639,"apply.run_wall_time_us":647,"num_ops":2,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":749,"raft.queue_time_us":66,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":19242}]]}
I20240503 14:12:25.712492 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07354861ee6142d6b1fbe573397d3160" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.712679 28352 raft_consensus.cc:3037] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.713356 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_851943f2c8df437dbfc6a677f5ef4900"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:25.713691 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab077988364bd687e54f90c3c265d5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.713891 28350 raft_consensus.cc:2454] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:25.714179 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_851943f2c8df437dbfc6a677f5ef4900 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.711921 28614 consensus_queue.cc:260] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.714709 28614 ts_tablet_manager.cc:1398] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.052s user 0.001s sys 0.001s
I20240503 14:12:25.715037 28614 tablet_bootstrap.cc:492] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:25.716722 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab077988364bd687e54f90c3c265d5" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.716917 28350 raft_consensus.cc:3037] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.718323 28352 raft_consensus.cc:2454] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:25.725898 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 5de9edfb0d064939bda2ec7a9e0e500b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:25.728233 29828 ts_tablet_manager.cc:1888] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:25.728385 29828 log.cc:1196] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/2221218847554cbd9d3242a9a6431e46
I20240503 14:12:25.730615 28614 tablet_bootstrap.cc:654] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.731233 28350 raft_consensus.cc:2454] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:25.731420 29828 ts_tablet_manager.cc:1909] T 2221218847554cbd9d3242a9a6431e46 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:25.731875 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 942) took 1018 ms. Trace:
I20240503 14:12:25.732849 29828 rpcz_store.cc:276] 0503 14:12:24.713023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.713063 (+ 40us) service_pool.cc:225] Handling call
0503 14:12:25.556864 (+843801us) ts_tablet_manager.cc:1120] Deleting tablet 2221218847554cbd9d3242a9a6431e46
0503 14:12:25.556875 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.664666 (+107791us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.728067 (+ 63401us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.731850 (+ 3783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":843663}
I20240503 14:12:25.737084 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 07354861ee6142d6b1fbe573397d3160 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.737541 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.737613 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbcae22ea124edaa14c9409f3052714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.734488 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75036cdf9e04ea4aa767fd12016f177" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.737766 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.737847 28350 raft_consensus.cc:2454] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:25.737871 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1958dc488e4854a8c8e74da9fab243 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.737920 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fad4508a29d4822b81af6947b57e7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.738060 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.738175 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.738286 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 1eb6d931156b492e82140ee342cb1bbb) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.738183 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 07354861ee6142d6b1fbe573397d3160 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.738762 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75036cdf9e04ea4aa767fd12016f177" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.739022 28355 raft_consensus.cc:3037] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.740041 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 943) took 1026 ms. Trace:
I20240503 14:12:25.742049 28355 raft_consensus.cc:2454] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:25.744761 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.745505 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ab077988364bd687e54f90c3c265d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cd037e135904e2cab09cbf39fed0b13) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.640491 28600 tablet_bootstrap.cc:654] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.754053 29996 consensus_queue.cc:1035] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.759235 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 77c77a11ef384709aef5b62273380573 (table default.Table_ab495b10ad82425ba5c7123a6e1cf064 [id=2edd1164c07a4a67b51262d402d53596]). Expected version 2 got 0
W20240503 14:12:25.761037 28427 leader_election.cc:341] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 903f11a608b1483887d9b5234a404470
I20240503 14:12:25.761209 28427 leader_election.cc:302] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:25.761372 28427 raft_consensus.cc:2670] T 903f11a608b1483887d9b5234a404470 P d472d9f94d8949cf86284d32a161cebc: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:25.767781 30134 consensus_queue.cc:1035] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.769620 28604 tablet_bootstrap.cc:492] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.769776 28604 ts_tablet_manager.cc:1367] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.085s user 0.001s sys 0.001s
I20240503 14:12:25.771176 28427 leader_election.cc:302] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:25.771322 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a27d21de6a6441d810f379bba88f9a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 0311cfbb060e439d9b7c497ffe4626cd) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.772440 28196 tablet_service.cc:1467] Processing CreateTablet for tablet acb304e0a391438cb7a430eca6b2d05d (DEFAULT_TABLE table=default.Table_2c1e6032d7f8400d84b575d7c1c08e32 [id=712273b6db8549fba3c876ae6486c1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.765064 30102 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0690a516c89446d85c88cdf64fa4238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } } }
I20240503 14:12:25.772892 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb304e0a391438cb7a430eca6b2d05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.772975 30102 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:25.776510 30134 raft_consensus.cc:2726] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.777017 28600 tablet_bootstrap.cc:492] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.777175 28600 ts_tablet_manager.cc:1367] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.000s
I20240503 14:12:25.778008 28600 raft_consensus.cc:348] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.778223 28600 raft_consensus.cc:374] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.778298 28600 raft_consensus.cc:729] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.778383 28427 leader_election.cc:302] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:25.778641 30134 raft_consensus.cc:2781] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.778550 28600 consensus_queue.cc:260] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.778759 30134 raft_consensus.cc:686] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:25.779047 30134 consensus_queue.cc:237] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.781244 28600 ts_tablet_manager.cc:1398] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:25.781594 28600 tablet_bootstrap.cc:492] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.783421 28600 tablet_bootstrap.cc:654] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.790297 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1958dc488e4854a8c8e74da9fab243 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.790906 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet d93fea5caec94695a5df26d3630c50c7 with delete_type TABLET_DATA_DELETED (replaced by tablet b476e87faea1415d95ae014a93fa22af) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.791464 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 79a96d73b2d142f0b6ada0c1c1cc654e (DEFAULT_TABLE table=default.Table_c1936a618b9f488fbd38c45bbc95303c [id=1654ff04429e4f1397b299342743bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.791961 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a96d73b2d142f0b6ada0c1c1cc654e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.792002 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 495a653a29494f59a8722901ba889d80 (DEFAULT_TABLE table=default.Table_f396f2a405c64cb98d09f08fedccd872 [id=074e2559b0134532bc0549b59b02ab0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.792441 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495a653a29494f59a8722901ba889d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.792376 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
I20240503 14:12:25.792693 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet e94673f00cb3432185009bda2ce398fa (table default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d [id=2849f6ba9ef64d26a5dc65f9ac9048da]). Expected version 1 got 0
I20240503 14:12:25.792881 28031 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:25.784716 28604 raft_consensus.cc:348] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.794060 28604 raft_consensus.cc:374] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.794166 28604 raft_consensus.cc:729] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.794777 28604 consensus_queue.cc:260] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.796135 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0dd46679b34bec8a1572490b5d2d0a (DEFAULT_TABLE table=default.Table_c9669afd79724f9e9f03dbcd90efab74 [id=96a0e8e5789d48d3b696de2a0f768bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.796516 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0dd46679b34bec8a1572490b5d2d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.796700 28604 ts_tablet_manager.cc:1398] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.027s user 0.000s sys 0.003s
I20240503 14:12:25.797072 28604 tablet_bootstrap.cc:492] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.798702 28604 tablet_bootstrap.cc:654] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.799158 30132 raft_consensus.cc:482] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.799309 30132 raft_consensus.cc:504] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.799695 28661 ts_tablet_manager.cc:1888] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:25.799795 28661 log.cc:1196] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/495190fca06a40a386c8215eb2195920
I20240503 14:12:25.800130 28661 ts_tablet_manager.cc:1909] T 495190fca06a40a386c8215eb2195920 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:25.800217 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 1eb6d931156b492e82140ee342cb1bbb) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.800022 30132 leader_election.cc:288] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.800419 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.800568 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:25.800858 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a27d21de6a6441d810f379bba88f9a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 0311cfbb060e439d9b7c497ffe4626cd) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:25.801282 28461 tablet_service.cc:1467] Processing CreateTablet for tablet be745429a5784269890f7e89dd2e1a8b (DEFAULT_TABLE table=default.Table_59522dec1afc43cbb814f4ecb19d9a16 [id=7a9e0eb0e127426298379e172207f77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.801647 28466 tablet_service.cc:1467] Processing CreateTablet for tablet acb304e0a391438cb7a430eca6b2d05d (DEFAULT_TABLE table=default.Table_2c1e6032d7f8400d84b575d7c1c08e32 [id=712273b6db8549fba3c876ae6486c1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.801796 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be745429a5784269890f7e89dd2e1a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.802009 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb304e0a391438cb7a430eca6b2d05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.804414 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0dd46679b34bec8a1572490b5d2d0a (DEFAULT_TABLE table=default.Table_c9669afd79724f9e9f03dbcd90efab74 [id=96a0e8e5789d48d3b696de2a0f768bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.804792 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0dd46679b34bec8a1572490b5d2d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.805732 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 851) took 4550 ms. Trace:
I20240503 14:12:25.805826 28661 rpcz_store.cc:276] 0503 14:12:21.255250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.265613 (+ 10363us) service_pool.cc:225] Handling call
0503 14:12:25.528325 (+4262712us) ts_tablet_manager.cc:1120] Deleting tablet 495190fca06a40a386c8215eb2195920
0503 14:12:25.528338 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.687766 (+159428us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.799594 (+111828us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.805706 (+ 6112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":4262481}
I20240503 14:12:25.806241 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 867) took 4406 ms. Trace:
I20240503 14:12:25.806314 28661 rpcz_store.cc:276] 0503 14:12:21.399401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.399447 (+ 46us) service_pool.cc:225] Handling call
0503 14:12:25.806116 (+4406669us) ts_tablet_manager.cc:1120] Deleting tablet b8d8f09130c04698949adc45f8b071d7
0503 14:12:25.806125 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.806225 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4245380}
I20240503 14:12:25.806604 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 868) took 4407 ms. Trace:
I20240503 14:12:25.806671 28661 rpcz_store.cc:276] 0503 14:12:21.399494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.402059 (+ 2565us) service_pool.cc:225] Handling call
0503 14:12:25.806512 (+4404453us) ts_tablet_manager.cc:1120] Deleting tablet 37dac592c9dd4888a2ce89fa922f7fc8
0503 14:12:25.806520 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.806591 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4242291}
I20240503 14:12:25.806957 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 870) took 4404 ms. Trace:
I20240503 14:12:25.807030 28661 rpcz_store.cc:276] 0503 14:12:21.402582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.403391 (+ 809us) service_pool.cc:225] Handling call
0503 14:12:25.806866 (+4403475us) ts_tablet_manager.cc:1120] Deleting tablet 927d4359637f4ec189a5143e8201c7f1
0503 14:12:25.806874 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.806944 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4239588}
I20240503 14:12:25.807282 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 871) took 4403 ms. Trace:
I20240503 14:12:25.807348 28661 rpcz_store.cc:276] 0503 14:12:21.403695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.404046 (+ 351us) service_pool.cc:225] Handling call
0503 14:12:25.807203 (+4403157us) ts_tablet_manager.cc:1120] Deleting tablet e8f9f10348014293a421ed11a50f89e7
0503 14:12:25.807211 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.807270 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4239852}
I20240503 14:12:25.807972 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:25.808626 28150 leader_election.cc:341] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.808750 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 79a96d73b2d142f0b6ada0c1c1cc654e (DEFAULT_TABLE table=default.Table_c1936a618b9f488fbd38c45bbc95303c [id=1654ff04429e4f1397b299342743bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.809118 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a96d73b2d142f0b6ada0c1c1cc654e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.820050 28604 tablet_bootstrap.cc:492] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.820145 30132 raft_consensus.cc:482] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.820240 28604 ts_tablet_manager.cc:1367] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20240503 14:12:25.820299 30132 raft_consensus.cc:504] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.820960 30132 leader_election.cc:288] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.821023 28604 raft_consensus.cc:348] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.821202 28604 raft_consensus.cc:374] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.821257 28604 raft_consensus.cc:729] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.821455 28604 consensus_queue.cc:260] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.821971 28661 tablet_replica.cc:330] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:25.822019 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a703cfbaad2a4a618abb9e07b4930cee" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.822173 28661 raft_consensus.cc:2229] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:25.823554 28661 raft_consensus.cc:2258] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:25.823637 28150 leader_election.cc:341] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.824129 28661 ts_tablet_manager.cc:1875] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.827204 30142 log.cc:1196] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e4a544fc27624dcc8a4cb003e59408f2
I20240503 14:12:25.827613 30142 ts_tablet_manager.cc:1909] T e4a544fc27624dcc8a4cb003e59408f2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:25.829211 30142 tablet_replica.cc:330] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:25.829640 30142 raft_consensus.cc:2229] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:25.830006 30142 raft_consensus.cc:2258] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.830799 30142 ts_tablet_manager.cc:1875] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.830960 28604 ts_tablet_manager.cc:1398] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 14:12:25.832017 28196 tablet_service.cc:1467] Processing CreateTablet for tablet be745429a5784269890f7e89dd2e1a8b (DEFAULT_TABLE table=default.Table_59522dec1afc43cbb814f4ecb19d9a16 [id=7a9e0eb0e127426298379e172207f77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.832448 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be745429a5784269890f7e89dd2e1a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.832665 28600 tablet_bootstrap.cc:492] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.832810 28600 ts_tablet_manager.cc:1367] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240503 14:12:25.833325 28604 tablet_bootstrap.cc:492] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.833652 28600 raft_consensus.cc:348] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.833932 28600 raft_consensus.cc:374] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.834002 28600 raft_consensus.cc:729] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.834280 28600 consensus_queue.cc:260] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.834900 28604 tablet_bootstrap.cc:654] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.835904 28600 ts_tablet_manager.cc:1398] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:25.836560 28600 tablet_bootstrap.cc:492] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:25.845301 28600 tablet_bootstrap.cc:654] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.847532 30132 raft_consensus.cc:482] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.847694 30132 raft_consensus.cc:504] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.848371 30132 leader_election.cc:288] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.740172 29828 rpcz_store.cc:276] 0503 14:12:24.713319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.713358 (+ 39us) service_pool.cc:225] Handling call
0503 14:12:25.739921 (+1026563us) ts_tablet_manager.cc:1120] Deleting tablet 77f0081cc2cb48aaa368741109032fd1
0503 14:12:25.739934 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.740017 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026430}
I20240503 14:12:25.849160 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.849258 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.849381 28487 raft_consensus.cc:2454] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.849421 29828 tablet_replica.cc:330] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:25.849624 29828 raft_consensus.cc:2229] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:25.849848 29828 raft_consensus.cc:2258] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:25.849858 28150 leader_election.cc:341] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.850127 28150 leader_election.cc:302] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:25.850584 30132 raft_consensus.cc:2781] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.850666 30132 raft_consensus.cc:482] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.850724 30132 raft_consensus.cc:3037] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.850850 29828 ts_tablet_manager.cc:1875] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:25.763717 28290 leader_election.cc:302] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:25.852273 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 88596.6us
I20240503 14:12:25.852494 28290 leader_election.cc:302] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.770596 28614 tablet_bootstrap.cc:492] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:25.782924 30169 tablet.cc:1722] T 3f1aa939aced4556ae0f0de0e62d1f97 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240503 14:12:25.853406 28614 ts_tablet_manager.cc:1367] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.003s
I20240503 14:12:25.853403 30132 raft_consensus.cc:504] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.822341 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a703cfbaad2a4a618abb9e07b4930cee" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.854272 30132 leader_election.cc:288] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.854460 28614 raft_consensus.cc:348] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.854648 28614 raft_consensus.cc:374] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.854702 28614 raft_consensus.cc:729] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.854998 28614 consensus_queue.cc:260] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.855190 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:25.855403 28487 raft_consensus.cc:3037] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.855453 29949 raft_consensus.cc:2781] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.855701 30070 raft_consensus.cc:2781] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.855775 30070 raft_consensus.cc:482] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.855834 30070 raft_consensus.cc:3037] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.822957 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.858197 28487 raft_consensus.cc:2454] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:25.858389 30070 raft_consensus.cc:504] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.859117 30070 leader_election.cc:288] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.859458 30070 consensus_queue.cc:1035] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.859854 30070 consensus_queue.cc:1035] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.861128 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 9964b86eabfd429c8336105c2f656e35 (DEFAULT_TABLE table=default.Table_52ef5212fbac426bac3aefbc92d9de91 [id=865f487fc71b441e93237c8679deed20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.861128 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 495a653a29494f59a8722901ba889d80 (DEFAULT_TABLE table=default.Table_f396f2a405c64cb98d09f08fedccd872 [id=074e2559b0134532bc0549b59b02ab0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.861600 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9964b86eabfd429c8336105c2f656e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.861881 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495a653a29494f59a8722901ba889d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.865032 29994 raft_consensus.cc:482] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.865164 29994 raft_consensus.cc:504] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.865360 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e4a544fc27624dcc8a4cb003e59408f2 (table default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]) successfully deleted
I20240503 14:12:25.865340 28150 leader_election.cc:302] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:25.865650 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2946ba624b244db9a617f2e202bfc1ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2946ba624b244db9a617f2e202bfc1ec already in progress: creating tablet
W20240503 14:12:25.865718 28150 leader_election.cc:341] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.865816 28150 leader_election.cc:302] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.865873 29994 leader_election.cc:288] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.866020 29811 raft_consensus.cc:2781] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.866130 29811 raft_consensus.cc:686] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
W20240503 14:12:25.866199 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7b89711872ba442ca66e35091a2271da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b89711872ba442ca66e35091a2271da already in progress: creating tablet
I20240503 14:12:25.866366 30129 raft_consensus.cc:2726] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.866436 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet c5297664a4d943c28adfb6ac4f061680 (table default.Table_78145a3b2c8543b5ab5a675ab4b08c91 [id=99ed67dc285f4dc69a9a9c6cec10b729]) successfully deleted
I20240503 14:12:25.866616 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 2221218847554cbd9d3242a9a6431e46 (table default.Table_4929b450373c4b689562f4a1d9e4b4f2 [id=7b85025507014faa9ce25ba44310b973]) successfully deleted
W20240503 14:12:25.866628 28150 leader_election.cc:341] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.866719 28150 leader_election.cc:302] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.866744 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd8fb8ea1a4fc3bfa4f55ea59ae12a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:25.866950 28215 raft_consensus.cc:2454] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:25.867118 30132 raft_consensus.cc:2726] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.866385 29811 consensus_queue.cc:237] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.872224 28427 leader_election.cc:302] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.872282 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 9964b86eabfd429c8336105c2f656e35 (DEFAULT_TABLE table=default.Table_52ef5212fbac426bac3aefbc92d9de91 [id=865f487fc71b441e93237c8679deed20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.872682 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9964b86eabfd429c8336105c2f656e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.874291 29994 raft_consensus.cc:2781] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.874379 29994 raft_consensus.cc:482] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.874433 29994 raft_consensus.cc:3037] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.879283 29994 raft_consensus.cc:504] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.880285 29994 leader_election.cc:288] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:25.880977 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd8fb8ea1a4fc3bfa4f55ea59ae12a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:25.881175 28215 raft_consensus.cc:3037] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.883394 29828 ts_tablet_manager.cc:1888] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:25.883527 29828 log.cc:1196] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/495190fca06a40a386c8215eb2195920
I20240503 14:12:25.883697 28215 raft_consensus.cc:2454] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:25.883908 29828 ts_tablet_manager.cc:1909] T 495190fca06a40a386c8215eb2195920 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:25.887495 30132 raft_consensus.cc:482] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.887733 30132 raft_consensus.cc:504] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:25.888515 30132 leader_election.cc:288] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.889578 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 944) took 1176 ms. Trace:
I20240503 14:12:25.889540 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d54cc5c03034a6685c1da34f9bd65ac" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.889670 29828 rpcz_store.cc:276] 0503 14:12:24.713593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.713629 (+ 36us) service_pool.cc:225] Handling call
0503 14:12:25.849360 (+1135731us) ts_tablet_manager.cc:1120] Deleting tablet 495190fca06a40a386c8215eb2195920
0503 14:12:25.849372 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.865726 (+ 16354us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.883259 (+ 17533us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.889549 (+ 6290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135599}
W20240503 14:12:25.890046 28150 leader_election.cc:341] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.890054 28427 leader_election.cc:302] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.890530 30134 raft_consensus.cc:2781] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.890669 30134 raft_consensus.cc:686] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:25.890545 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d54cc5c03034a6685c1da34f9bd65ac" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.890913 30134 consensus_queue.cc:237] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:25.891458 28150 leader_election.cc:341] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.891619 28150 leader_election.cc:302] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.891687 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.892279 30129 raft_consensus.cc:2726] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.898978 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 495190fca06a40a386c8215eb2195920 (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]) successfully deleted
W20240503 14:12:25.899127 28150 leader_election.cc:341] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:25.899618 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b8d8f09130c04698949adc45f8b071d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d8f09130c04698949adc45f8b071d7 already in progress: creating tablet
I20240503 14:12:25.899716 29811 raft_consensus.cc:482] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.899869 29811 raft_consensus.cc:504] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:25.900007 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 37dac592c9dd4888a2ce89fa922f7fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37dac592c9dd4888a2ce89fa922f7fc8 already in progress: creating tablet
W20240503 14:12:25.900288 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 927d4359637f4ec189a5143e8201c7f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 927d4359637f4ec189a5143e8201c7f1 already in progress: creating tablet
I20240503 14:12:25.900646 29811 leader_election.cc:288] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:25.900687 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e8f9f10348014293a421ed11a50f89e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8f9f10348014293a421ed11a50f89e7 already in progress: creating tablet
I20240503 14:12:25.901330 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.901371 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:25.901894 28150 leader_election.cc:341] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:25.902110 28150 leader_election.cc:341] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.895624 29828 tablet_replica.cc:330] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:25.902202 28150 leader_election.cc:302] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.902390 29828 raft_consensus.cc:2229] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:25.902587 29828 raft_consensus.cc:2258] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:25.904368 28614 ts_tablet_manager.cc:1398] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.051s user 0.003s sys 0.007s
I20240503 14:12:25.904716 28614 tablet_bootstrap.cc:492] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:25.905480 29949 raft_consensus.cc:686] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:25.905557 29811 raft_consensus.cc:2726] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.905910 29949 consensus_queue.cc:237] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.907814 29828 ts_tablet_manager.cc:1875] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:25.914265 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 61104.9us
I20240503 14:12:25.925609 28661 ts_tablet_manager.cc:1888] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.940562 28661 log.cc:1196] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/5a081ba59ad44890bf2f94874c034277
I20240503 14:12:25.941118 28661 ts_tablet_manager.cc:1909] T 5a081ba59ad44890bf2f94874c034277 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:25.942071 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 872) took 4537 ms. Trace:
I20240503 14:12:25.929353 30177 consensus_queue.cc:1035] T 3c61b9232ebd4431a3d4202ad971d540 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240503 14:12:25.942932 28026 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1119) took 1262 ms. Trace:
I20240503 14:12:25.943032 28026 rpcz_store.cc:276] 0503 14:12:24.680029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.681915 (+1001886us) service_pool.cc:225] Handling call
0503 14:12:25.682291 (+ 376us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:25.682477 (+ 186us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:25.682674 (+ 197us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:25.682764 (+ 90us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:25.942439 (+259675us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:25.942866 (+ 427us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.684072 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.684207 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.684214 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.684217 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.684326 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.684338 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.684342 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:25.684408 (+ 66us) tablet.cc:699] Row locks acquired
0503 14:12:25.684410 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.684447 (+ 37us) write_op.cc:270] Start()
0503 14:12:25.684476 (+ 29us) write_op.cc:276] Timestamp: P: 1714745545684422 usec, L: 0
0503 14:12:25.684480 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.684662 (+ 182us) log.cc:842] Serialized 405 byte log entry
0503 14:12:25.939957 (+255295us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.940481 (+ 524us) write_op.cc:301] APPLY: starting
0503 14:12:25.940505 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.940553 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.940583 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.940652 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.940693 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.940701 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.940817 (+ 116us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.940895 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.940920 (+ 25us) write_op.cc:454] Released schema lock
0503 14:12:25.940929 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.940993 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.940997 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":450,"apply.run_cpu_time_us":584,"apply.run_wall_time_us":584,"num_ops":1,"prepare.queue_time_us":796,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":680,"raft.queue_time_us":37,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":255446,"spinlock_wait_cycles":17280}]]}
I20240503 14:12:25.942544 28290 leader_election.cc:302] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:25.943958 30107 raft_consensus.cc:2726] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:25.928066 28614 tablet_bootstrap.cc:654] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.944823 28024 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1117) took 1389 ms. Trace:
I20240503 14:12:25.944947 28024 rpcz_store.cc:276] 0503 14:12:24.555821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.659578 (+1103757us) service_pool.cc:225] Handling call
0503 14:12:25.688546 (+ 28968us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.689016 (+ 470us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.944546 (+255530us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.944675 (+ 129us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.944778 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.689759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.689858 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.689865 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.689868 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.689977 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.689993 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.689999 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.690086 (+ 87us) tablet.cc:699] Row locks acquired
0503 14:12:25.690089 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.690134 (+ 45us) write_op.cc:270] Start()
0503 14:12:25.690174 (+ 40us) write_op.cc:276] Timestamp: P: 1714745545690106 usec, L: 0
0503 14:12:25.690179 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.690400 (+ 221us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.940038 (+249638us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.941092 (+ 1054us) write_op.cc:301] APPLY: starting
0503 14:12:25.941107 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.941146 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.941171 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.941290 (+ 119us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.941332 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.941340 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.941409 (+ 69us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.941460 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.941499 (+ 39us) write_op.cc:454] Released schema lock
0503 14:12:25.941505 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.941559 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.941563 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1020,"apply.run_cpu_time_us":520,"apply.run_wall_time_us":519,"num_ops":2,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":767,"prepare.run_wall_time_us":768,"raft.queue_time_us":45,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":249844}]]}
W20240503 14:12:25.946656 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dee5a230ef584077ba9f4a27c878ef4d consensus state has no leader
I20240503 14:12:25.946667 30142 ts_tablet_manager.cc:1888] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:25.946810 30142 log.cc:1196] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6d6b50aed01a4bf8af8889036dd83606
I20240503 14:12:25.946836 28025 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1121) took 1264 ms. Trace:
I20240503 14:12:25.947192 30142 ts_tablet_manager.cc:1909] T 6d6b50aed01a4bf8af8889036dd83606 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:25.946941 28025 rpcz_store.cc:276] 0503 14:12:24.682418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.691525 (+1009107us) service_pool.cc:225] Handling call
0503 14:12:25.691951 (+ 426us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:25.692285 (+ 334us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:25.692476 (+ 191us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:25.692562 (+ 86us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:25.946406 (+253844us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:25.946797 (+ 391us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.693476 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.693578 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.693584 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.693586 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.693686 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.693699 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.693704 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:25.693777 (+ 73us) tablet.cc:699] Row locks acquired
0503 14:12:25.693779 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.693822 (+ 43us) write_op.cc:270] Start()
0503 14:12:25.693856 (+ 34us) write_op.cc:276] Timestamp: P: 1714745545693795 usec, L: 0
0503 14:12:25.693860 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.694046 (+ 186us) log.cc:842] Serialized 405 byte log entry
0503 14:12:25.940103 (+246057us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.941643 (+ 1540us) write_op.cc:301] APPLY: starting
0503 14:12:25.941655 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.941685 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.941707 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.941745 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.941770 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.941776 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:25.941848 (+ 72us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.941887 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.941900 (+ 13us) write_op.cc:454] Released schema lock
0503 14:12:25.941906 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.941953 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.941956 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1509,"apply.run_cpu_time_us":355,"apply.run_wall_time_us":353,"num_ops":1,"prepare.queue_time_us":406,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":712,"raft.queue_time_us":70,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":246209,"threads_started":1}]]}
I20240503 14:12:25.947664 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 9f5b973ea16d42babd2cc75e32624854 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:25.948702 30107 consensus_queue.cc:1035] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.952589 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 2946ba624b244db9a617f2e202bfc1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.952894 28661 rpcz_store.cc:276] 0503 14:12:21.404336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.404642 (+ 306us) service_pool.cc:225] Handling call
0503 14:12:25.821917 (+4417275us) ts_tablet_manager.cc:1120] Deleting tablet 5a081ba59ad44890bf2f94874c034277
0503 14:12:25.821927 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.845980 (+ 24053us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.925478 (+ 79498us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:25.942046 (+ 16568us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1224,"tablet-delete.queue_time_us":4254509}
I20240503 14:12:25.954388 28614 tablet_bootstrap.cc:492] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:25.954555 28614 ts_tablet_manager.cc:1367] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20240503 14:12:25.955420 28614 raft_consensus.cc:348] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.955631 28614 raft_consensus.cc:374] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.955685 28614 raft_consensus.cc:729] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.955905 28614 consensus_queue.cc:260] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.950832 28604 tablet_bootstrap.cc:492] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:25.956938 28604 ts_tablet_manager.cc:1367] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.124s user 0.005s sys 0.003s
I20240503 14:12:25.957897 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 873) took 4553 ms. Trace:
I20240503 14:12:25.958755 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 5a081ba59ad44890bf2f94874c034277 (table default.Table_1145b045780c4d1e8fedd333eea661d4 [id=114dfd39b4ea4958a08161a34e4a1f9f]) successfully deleted
I20240503 14:12:25.959057 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b89711872ba442ca66e35091a2271da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.961592 28029 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1118) took 1403 ms. Trace:
I20240503 14:12:25.961715 28029 rpcz_store.cc:276] 0503 14:12:24.558506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.662832 (+1104326us) service_pool.cc:225] Handling call
0503 14:12:25.663879 (+ 1047us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.694939 (+ 31060us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.961278 (+266339us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.961422 (+ 144us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.961525 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.702633 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.702780 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.702788 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:25.702791 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.702900 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.702913 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.702919 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.703005 (+ 86us) tablet.cc:699] Row locks acquired
0503 14:12:25.703008 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.703060 (+ 52us) write_op.cc:270] Start()
0503 14:12:25.703102 (+ 42us) write_op.cc:276] Timestamp: P: 1714745545703030 usec, L: 0
0503 14:12:25.703106 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.703364 (+ 258us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.960224 (+256860us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.960368 (+ 144us) write_op.cc:301] APPLY: starting
0503 14:12:25.960386 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.960430 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.960457 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.960564 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.960598 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.960606 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.960683 (+ 77us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.960744 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.960764 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:25.960770 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.960824 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.960827 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":509,"apply.run_wall_time_us":508,"num_ops":2,"prepare.queue_time_us":7153,"prepare.run_cpu_time_us":875,"prepare.run_wall_time_us":875,"raft.queue_time_us":55,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":257091}]]}
I20240503 14:12:25.962702 29828 ts_tablet_manager.cc:1888] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:25.962817 29828 log.cc:1196] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e4a544fc27624dcc8a4cb003e59408f2
W20240503 14:12:25.962818 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 310abf35ae02497c9d3696a0c49774ae consensus state has no leader
I20240503 14:12:25.962988 28033 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1116) took 1422 ms. Trace:
I20240503 14:12:25.963121 29828 ts_tablet_manager.cc:1909] T e4a544fc27624dcc8a4cb003e59408f2 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:25.963083 28033 rpcz_store.cc:276] 0503 14:12:24.540421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.654996 (+1114575us) service_pool.cc:225] Handling call
0503 14:12:25.655380 (+ 384us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:25.655571 (+ 191us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:25.655783 (+ 212us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:25.655873 (+ 90us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:25.962560 (+306687us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:25.962951 (+ 391us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.703541 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.703817 (+ 276us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.703823 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.703826 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.703939 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.703950 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.703955 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:25.704020 (+ 65us) tablet.cc:699] Row locks acquired
0503 14:12:25.704023 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.704063 (+ 40us) write_op.cc:270] Start()
0503 14:12:25.704117 (+ 54us) write_op.cc:276] Timestamp: P: 1714745545704036 usec, L: 0
0503 14:12:25.704122 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.704309 (+ 187us) log.cc:842] Serialized 405 byte log entry
0503 14:12:25.960293 (+255984us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.960907 (+ 614us) write_op.cc:301] APPLY: starting
0503 14:12:25.960918 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.960947 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.960969 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.961003 (+ 34us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.961027 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.961034 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.961084 (+ 50us) log.cc:842] Serialized 23 byte log entry
0503 14:12:25.961120 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.961131 (+ 11us) write_op.cc:454] Released schema lock
0503 14:12:25.961135 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.961176 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.961179 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":588,"apply.run_cpu_time_us":305,"apply.run_wall_time_us":305,"num_ops":1,"prepare.queue_time_us":7829,"prepare.run_cpu_time_us":758,"prepare.run_wall_time_us":874,"raft.queue_time_us":51,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":256157}]]}
I20240503 14:12:25.964025 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet d93fea5caec94695a5df26d3630c50c7 with delete_type TABLET_DATA_DELETED (replaced by tablet b476e87faea1415d95ae014a93fa22af) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.964406 28661 rpcz_store.cc:276] 0503 14:12:21.404874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.405200 (+ 326us) service_pool.cc:225] Handling call
0503 14:12:25.957732 (+4552532us) ts_tablet_manager.cc:1120] Deleting tablet 3112560e16f14b4e89aae90c014be9fb
0503 14:12:25.957742 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.957875 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4390247}
I20240503 14:12:25.965318 28604 raft_consensus.cc:348] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.965714 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.966552 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.966986 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1958dc488e4854a8c8e74da9fab243 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.967056 28030 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1120) took 1284 ms. Trace:
I20240503 14:12:25.967170 28030 rpcz_store.cc:276] 0503 14:12:24.682317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.685280 (+1002963us) service_pool.cc:225] Handling call
0503 14:12:25.687382 (+ 2102us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.687894 (+ 512us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.966742 (+278848us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.966869 (+ 127us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.966970 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.709589 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.709696 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.709702 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.709705 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.709809 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.709820 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.709825 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.709960 (+ 135us) tablet.cc:699] Row locks acquired
0503 14:12:25.709963 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.710011 (+ 48us) write_op.cc:270] Start()
0503 14:12:25.710044 (+ 33us) write_op.cc:276] Timestamp: P: 1714745545709982 usec, L: 0
0503 14:12:25.710047 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.710340 (+ 293us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.966037 (+255697us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.966138 (+ 101us) write_op.cc:301] APPLY: starting
0503 14:12:25.966153 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.966192 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.966216 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.966337 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.966372 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.966380 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.966452 (+ 72us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.966517 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.966535 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:25.966542 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.966618 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.966623 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"apply.run_cpu_time_us":534,"apply.run_wall_time_us":534,"num_ops":2,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":785,"prepare.run_wall_time_us":1236,"raft.queue_time_us":49,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":255962}]]}
I20240503 14:12:25.967875 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 19d43cc864064c4c907ec0629391da10 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:25.970197 28027 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1122) took 1241 ms. Trace:
I20240503 14:12:25.970348 28027 rpcz_store.cc:276] 0503 14:12:24.728529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.713096 (+984567us) service_pool.cc:225] Handling call
0503 14:12:25.714453 (+ 1357us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.715075 (+ 622us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.969932 (+254857us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.970061 (+ 129us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.970157 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.716153 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.716264 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.716270 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:25.716272 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.716365 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.716377 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.716382 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.716461 (+ 79us) tablet.cc:699] Row locks acquired
0503 14:12:25.716463 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.716503 (+ 40us) write_op.cc:270] Start()
0503 14:12:25.716541 (+ 38us) write_op.cc:276] Timestamp: P: 1714745545716475 usec, L: 0
0503 14:12:25.716544 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.716759 (+ 215us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.969279 (+252520us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.969383 (+ 104us) write_op.cc:301] APPLY: starting
0503 14:12:25.969398 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.969438 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.969460 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.969564 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.969598 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.969605 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.969677 (+ 72us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.969738 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.969756 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:25.969762 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.969813 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.969816 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":480,"apply.run_wall_time_us":480,"num_ops":2,"prepare.queue_time_us":76,"prepare.run_cpu_time_us":726,"prepare.run_wall_time_us":1100,"raft.queue_time_us":40,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":252706}]]}
I20240503 14:12:25.972211 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbcae22ea124edaa14c9409f3052714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.972633 28604 raft_consensus.cc:374] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.972729 28604 raft_consensus.cc:729] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.966383 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 869) took 4563 ms. Trace:
I20240503 14:12:25.972916 28661 rpcz_store.cc:276] 0503 14:12:21.402481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.402624 (+ 143us) service_pool.cc:225] Handling call
0503 14:12:25.966236 (+4563612us) ts_tablet_manager.cc:1120] Deleting tablet 8097ba5606cf42d68448833de36d85c7
0503 14:12:25.966248 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.966363 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4398609}
I20240503 14:12:25.973070 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5de9edfb0d064939bda2ec7a9e0e500b
I20240503 14:12:25.973378 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cd09e769be4bee95bbef9c3c550451 with delete_type TABLET_DATA_DELETED (replaced by tablet 49445f0f1dd1451c9ba066e61273f2f1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.973572 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 5de9edfb0d064939bda2ec7a9e0e500b deleted and replaced by tablet 1906ab252ea349cfa0afbd9f4303efbc
I20240503 14:12:25.973709 28032 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1123) took 1235 ms. Trace:
I20240503 14:12:25.973781 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 880) took 4143 ms. Trace:
I20240503 14:12:25.973861 28661 rpcz_store.cc:276] 0503 14:12:21.830216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.844557 (+ 14341us) service_pool.cc:225] Handling call
0503 14:12:25.973627 (+4129070us) ts_tablet_manager.cc:1120] Deleting tablet 6874d970e4354070b11060c6b861d7bf
0503 14:12:25.973640 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.973758 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4128902}
I20240503 14:12:25.973788 28032 rpcz_store.cc:276] 0503 14:12:24.738468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.725717 (+987249us) service_pool.cc:225] Handling call
0503 14:12:25.973655 (+247938us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.727292 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.727424 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.727431 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.727433 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:25.727566 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.727581 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.727587 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.727675 (+ 88us) tablet.cc:699] Row locks acquired
0503 14:12:25.727677 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:25.727722 (+ 45us) write_op.cc:270] Start()
0503 14:12:25.727756 (+ 34us) write_op.cc:276] Timestamp: P: 1714745545727692 usec, L: 0
0503 14:12:25.727760 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.727969 (+ 209us) log.cc:842] Serialized 628 byte log entry
0503 14:12:25.972133 (+244164us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.972220 (+ 87us) write_op.cc:301] APPLY: starting
0503 14:12:25.972235 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.972273 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.972298 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.972413 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.972465 (+ 52us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:25.972473 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:25.972546 (+ 73us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.972610 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.972628 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:25.972634 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.972688 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.972691 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":519,"apply.run_wall_time_us":519,"num_ops":2,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":835,"prepare.run_wall_time_us":1402,"raft.queue_time_us":56,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":10,"replication_time_us":244346}]]}
I20240503 14:12:25.974438 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6b50aed01a4bf8af8889036dd83606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:08 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.974880 28604 consensus_queue.cc:260] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:25.975203 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fad4508a29d4822b81af6947b57e7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.975523 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe3affd0d144b1fb0078e7690b9b36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.976680 28604 ts_tablet_manager.cc:1398] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240503 14:12:25.976926 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d28d3c2ea472191ad21355248a2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.977770 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.978360 29811 raft_consensus.cc:482] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.978619 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 1eb6d931156b492e82140ee342cb1bbb) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.978557 29811 raft_consensus.cc:504] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.979080 28614 ts_tablet_manager.cc:1398] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240503 14:12:25.979353 29811 leader_election.cc:288] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.979538 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.976135 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 879) took 4145 ms. Trace:
I20240503 14:12:25.979818 28661 rpcz_store.cc:276] 0503 14:12:21.830141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.844496 (+ 14355us) service_pool.cc:225] Handling call
0503 14:12:25.975951 (+4131455us) ts_tablet_manager.cc:1120] Deleting tablet 752b369f24224f2486834814a9d08495
0503 14:12:25.975965 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.976110 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4126166}
I20240503 14:12:25.980491 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ac912a3bd64cdc9082596e09c72e13" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:25.980779 28487 raft_consensus.cc:2454] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.981137 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ac912a3bd64cdc9082596e09c72e13" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:25.981352 28355 raft_consensus.cc:2454] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:25.981418 28150 leader_election.cc:302] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:25.981918 28604 tablet_bootstrap.cc:492] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:25.981961 29811 raft_consensus.cc:2781] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.982048 29811 raft_consensus.cc:482] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.982107 29811 raft_consensus.cc:3037] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.980504 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 878) took 4150 ms. Trace:
I20240503 14:12:25.984176 28661 rpcz_store.cc:276] 0503 14:12:21.830075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.844195 (+ 14120us) service_pool.cc:225] Handling call
0503 14:12:25.980338 (+4136143us) ts_tablet_manager.cc:1120] Deleting tablet 281667db9a9c43d3b9d7dd9ee14b4b06
0503 14:12:25.980351 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.980483 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4130353}
I20240503 14:12:25.984874 29811 raft_consensus.cc:504] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.986007 29811 leader_election.cc:288] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:25.986227 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 883) took 4155 ms. Trace:
I20240503 14:12:25.986322 28661 rpcz_store.cc:276] 0503 14:12:21.830413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.844818 (+ 14405us) service_pool.cc:225] Handling call
0503 14:12:25.984655 (+4139837us) ts_tablet_manager.cc:1120] Deleting tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4
0503 14:12:25.986085 (+ 1430us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.986205 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4097072}
I20240503 14:12:25.986716 28028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1115) took 1446 ms. Trace:
I20240503 14:12:25.986826 28028 rpcz_store.cc:276] 0503 14:12:24.540322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.635684 (+1095362us) service_pool.cc:225] Handling call
0503 14:12:25.640716 (+ 5032us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:25.643187 (+ 2471us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:25.986407 (+343220us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:25.986533 (+ 126us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:25.986644 (+ 111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:25.748594 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:25.748723 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:25.748730 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:25.748733 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:25.748835 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:25.748849 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:25.748854 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:25.748941 (+ 87us) tablet.cc:699] Row locks acquired
0503 14:12:25.748944 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:25.750397 (+ 1453us) write_op.cc:270] Start()
0503 14:12:25.750444 (+ 47us) write_op.cc:276] Timestamp: P: 1714745545748955 usec, L: 0
0503 14:12:25.750449 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:25.750666 (+ 217us) log.cc:842] Serialized 433 byte log entry
0503 14:12:25.984046 (+233380us) op_driver.cc:444] REPLICATION: finished
0503 14:12:25.985801 (+ 1755us) write_op.cc:301] APPLY: starting
0503 14:12:25.985821 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:25.985866 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:25.985892 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:25.985998 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:25.986031 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:25.986038 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:25.986116 (+ 78us) log.cc:842] Serialized 29 byte log entry
0503 14:12:25.986181 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.986201 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:25.986208 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:25.986267 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:25.986271 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1679,"apply.run_cpu_time_us":534,"apply.run_wall_time_us":534,"num_ops":2,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":831,"prepare.run_wall_time_us":3661,"raft.queue_time_us":1048,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":233564}]]}
I20240503 14:12:25.987381 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ac912a3bd64cdc9082596e09c72e13" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:25.987592 28355 raft_consensus.cc:3037] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.989112 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410e8b6b511403ea609b1353855937b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:25.990084 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_33b986666be7452c8edcd5b0a0bf4f6f" } modify_external_catalogs: true
I20240503 14:12:25.990319 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_33b986666be7452c8edcd5b0a0bf4f6f" } modify_external_catalogs: true
I20240503 14:12:25.990360 28355 raft_consensus.cc:2454] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:25.991041 28150 leader_election.cc:302] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:25.991196 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6349dd55828a4c548a0472ada6ffdc73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:25.991822 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6349dd55828a4c548a0472ada6ffdc73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.991865 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 882) took 4161 ms. Trace:
I20240503 14:12:25.991812 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11ac912a3bd64cdc9082596e09c72e13" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:25.992024 28487 raft_consensus.cc:3037] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.992763 28604 tablet_bootstrap.cc:654] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.991945 28661 rpcz_store.cc:276] 0503 14:12:21.830351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.844795 (+ 14444us) service_pool.cc:225] Handling call
0503 14:12:25.986720 (+4141925us) ts_tablet_manager.cc:1120] Deleting tablet 60d4f5591b1f4826bc09e57094fb46bf
0503 14:12:25.991688 (+ 4968us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.991843 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4098416}
I20240503 14:12:25.994583 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0573272ac5c84afc82be96eea79f4ce3" } modify_external_catalogs: true
I20240503 14:12:25.994827 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_0573272ac5c84afc82be96eea79f4ce3" } modify_external_catalogs: true
I20240503 14:12:25.994916 28487 raft_consensus.cc:2454] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
W20240503 14:12:25.995368 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 3112560e16f14b4e89aae90c014be9fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3112560e16f14b4e89aae90c014be9fb already in progress: creating tablet
I20240503 14:12:25.995574 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9edfb0d064939bda2ec7a9e0e500b with delete_type TABLET_DATA_DELETED (replaced by tablet 1906ab252ea349cfa0afbd9f4303efbc) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:25.995738 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6874d970e4354070b11060c6b861d7bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6874d970e4354070b11060c6b861d7bf already in progress: creating tablet
W20240503 14:12:25.995977 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 752b369f24224f2486834814a9d08495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 752b369f24224f2486834814a9d08495 already in progress: creating tablet
I20240503 14:12:25.996158 29811 raft_consensus.cc:2781] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
W20240503 14:12:25.996210 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 281667db9a9c43d3b9d7dd9ee14b4b06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 281667db9a9c43d3b9d7dd9ee14b4b06 already in progress: creating tablet
W20240503 14:12:25.996515 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 60d4f5591b1f4826bc09e57094fb46bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d4f5591b1f4826bc09e57094fb46bf already in progress: creating tablet
I20240503 14:12:25.996465 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9dfb63e159c34cb9a11a9a5976aa4979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:25.996867 29811 raft_consensus.cc:686] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
W20240503 14:12:25.997040 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dfb63e159c34cb9a11a9a5976aa4979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:25.997339 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 881) took 4167 ms. Trace:
I20240503 14:12:25.997294 29811 consensus_queue.cc:237] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.997416 28661 rpcz_store.cc:276] 0503 14:12:21.830290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.844764 (+ 14474us) service_pool.cc:225] Handling call
0503 14:12:25.994080 (+4149316us) ts_tablet_manager.cc:1120] Deleting tablet d952b436198141ad80fef5fb1ea1e736
0503 14:12:25.997187 (+ 3107us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.997314 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4090264}
I20240503 14:12:25.998340 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 877) took 4168 ms. Trace:
I20240503 14:12:25.998421 28661 rpcz_store.cc:276] 0503 14:12:21.830000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:21.847850 (+ 17850us) service_pool.cc:225] Handling call
0503 14:12:25.997835 (+4149985us) ts_tablet_manager.cc:1120] Deleting tablet 10876f4512794e6aa72bda3c1da8b2d0
0503 14:12:25.998211 (+ 376us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:25.998321 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4092358}
I20240503 14:12:25.999699 28600 tablet_bootstrap.cc:492] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:25.999734 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4afe0b312f964d3885779a752ec8c256" } new_table_name: "default.Table_ca6cee392d934dd3ac773e100ae031d2" modify_external_catalogs: true
I20240503 14:12:25.999967 28600 ts_tablet_manager.cc:1367] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.163s user 0.007s sys 0.006s
I20240503 14:12:26.000770 28600 raft_consensus.cc:348] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:25.996894 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9edfb0d064939bda2ec7a9e0e500b with delete_type TABLET_DATA_DELETED (replaced by tablet 1906ab252ea349cfa0afbd9f4303efbc) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.001644 28661 tablet_replica.cc:330] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.001904 28661 raft_consensus.cc:2229] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.002259 28661 raft_consensus.cc:2258] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.002867 28661 ts_tablet_manager.cc:1875] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.013092 28614 tablet_bootstrap.cc:492] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:26.014926 28614 tablet_bootstrap.cc:654] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.015906 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6202333af943cbbc0d09553c1c193c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.016388 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet bf21f6234dd64cf8a32f1599fe50746e with delete_type TABLET_DATA_DELETED (replaced by tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.016736 30142 tablet_replica.cc:330] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.016755 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet d43823da09bb49be8f5d1a1149a4b11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.017063 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 920be6d87c4440d0bf7453c86cfe0b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.017125 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6d6b50aed01a4bf8af8889036dd83606 (table default.Table_ede6e833710649e08e31516f0890c2d6 [id=3d49172b73ba44538840b651a85b24e3]) successfully deleted
I20240503 14:12:26.017354 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bba6a3d6e84552b027f23a29cbb38a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:26.017460 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d952b436198141ad80fef5fb1ea1e736 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d952b436198141ad80fef5fb1ea1e736 already in progress: creating tablet
I20240503 14:12:26.017632 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6c998e518541248a87e4c1551ec080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:26.017695 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 10876f4512794e6aa72bda3c1da8b2d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10876f4512794e6aa72bda3c1da8b2d0 already in progress: creating tablet
I20240503 14:12:26.017925 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.018199 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a27d21de6a6441d810f379bba88f9a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 0311cfbb060e439d9b7c497ffe4626cd) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.018708 28318 tablet_service.cc:1467] Processing CreateTablet for tablet acb304e0a391438cb7a430eca6b2d05d (DEFAULT_TABLE table=default.Table_2c1e6032d7f8400d84b575d7c1c08e32 [id=712273b6db8549fba3c876ae6486c1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.019125 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb304e0a391438cb7a430eca6b2d05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.019780 28604 tablet_bootstrap.cc:492] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.020025 28604 ts_tablet_manager.cc:1367] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240503 14:12:26.020458 30142 raft_consensus.cc:2229] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.020895 28604 raft_consensus.cc:348] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.021137 28604 raft_consensus.cc:374] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.021251 28604 raft_consensus.cc:729] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.021569 28604 consensus_queue.cc:260] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.023402 28604 ts_tablet_manager.cc:1398] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:26.024724 28604 tablet_bootstrap.cc:492] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.001039 28600 raft_consensus.cc:374] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.026412 28600 raft_consensus.cc:729] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.026787 28600 consensus_queue.cc:260] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.028543 30142 raft_consensus.cc:2258] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.029978 30142 ts_tablet_manager.cc:1875] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.032711 28600 ts_tablet_manager.cc:1398] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240503 14:12:26.033105 28600 tablet_bootstrap.cc:492] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.034754 28600 tablet_bootstrap.cc:654] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.039268 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1fafcfc2a1b747f0a7047b61caefb5df" } new_table_name: "default.Table_16dc72b61e7248c191c1daf2d14f63a9" modify_external_catalogs: true
W20240503 14:12:26.044142 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e7b2865e9604ce58280321c7ee2a88e consensus state has no leader
I20240503 14:12:26.044639 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d97ddf0257014cd7835a63e1730e6907" } modify_external_catalogs: true
I20240503 14:12:26.044835 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d97ddf0257014cd7835a63e1730e6907" } modify_external_catalogs: true
I20240503 14:12:26.046973 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3307e69a5d4f4765b6ffc39e9df66fa1" } modify_external_catalogs: true
I20240503 14:12:26.047180 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3307e69a5d4f4765b6ffc39e9df66fa1" } modify_external_catalogs: true
I20240503 14:12:26.049348 28604 tablet_bootstrap.cc:654] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:26.054615 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f5b973ea16d42babd2cc75e32624854 consensus state has no leader
I20240503 14:12:26.055599 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_de7c489221b549229264191fa22fffac" } modify_external_catalogs: true
I20240503 14:12:26.055851 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_de7c489221b549229264191fa22fffac" } modify_external_catalogs: true
I20240503 14:12:26.057626 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 79a96d73b2d142f0b6ada0c1c1cc654e (DEFAULT_TABLE table=default.Table_c1936a618b9f488fbd38c45bbc95303c [id=1654ff04429e4f1397b299342743bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.057649 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0dd46679b34bec8a1572490b5d2d0a (DEFAULT_TABLE table=default.Table_c9669afd79724f9e9f03dbcd90efab74 [id=96a0e8e5789d48d3b696de2a0f768bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.058193 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a96d73b2d142f0b6ada0c1c1cc654e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.058616 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 495a653a29494f59a8722901ba889d80 (DEFAULT_TABLE table=default.Table_f396f2a405c64cb98d09f08fedccd872 [id=074e2559b0134532bc0549b59b02ab0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.058987 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495a653a29494f59a8722901ba889d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.059937 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0dd46679b34bec8a1572490b5d2d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.063119 28661 ts_tablet_manager.cc:1888] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:26.063349 28661 log.cc:1196] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/dda39f0323ec47799d3d44c96f3ba605
I20240503 14:12:26.063809 28661 ts_tablet_manager.cc:1909] T dda39f0323ec47799d3d44c96f3ba605 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.064442 30193 tablet.cc:1722] T df63f75c157a4e928d4dbbda06cfa5ce P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:26.065296 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 890) took 3368 ms. Trace:
I20240503 14:12:26.065377 30194 tablet.cc:1722] T df63f75c157a4e928d4dbbda06cfa5ce P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:26.071342 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 9964b86eabfd429c8336105c2f656e35 (DEFAULT_TABLE table=default.Table_52ef5212fbac426bac3aefbc92d9de91 [id=865f487fc71b441e93237c8679deed20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.071974 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9964b86eabfd429c8336105c2f656e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.074404 30142 ts_tablet_manager.cc:1888] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.074375 28318 tablet_service.cc:1467] Processing CreateTablet for tablet be745429a5784269890f7e89dd2e1a8b (DEFAULT_TABLE table=default.Table_59522dec1afc43cbb814f4ecb19d9a16 [id=7a9e0eb0e127426298379e172207f77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.074770 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be745429a5784269890f7e89dd2e1a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.076216 28661 rpcz_store.cc:276] 0503 14:12:22.697195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.697253 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:25.998772 (+3301519us) ts_tablet_manager.cc:1120] Deleting tablet dda39f0323ec47799d3d44c96f3ba605
0503 14:12:26.001586 (+ 2814us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.006233 (+ 4647us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.063011 (+ 56778us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.065270 (+ 2259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3214138}
I20240503 14:12:26.066226 28604 tablet_bootstrap.cc:492] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.076723 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet dda39f0323ec47799d3d44c96f3ba605 (table default.Table_6c623db75c5548ebb88cdc9a786d0555 [id=8b305f063c044901b54e74d473d73cb8]) successfully deleted
I20240503 14:12:26.076810 30142 log.cc:1196] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/8c6202333af943cbbc0d09553c1c193c
I20240503 14:12:26.076968 28604 ts_tablet_manager.cc:1367] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20240503 14:12:26.077322 30142 ts_tablet_manager.cc:1909] T 8c6202333af943cbbc0d09553c1c193c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.077821 28604 raft_consensus.cc:348] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.080262 29828 tablet_replica.cc:330] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:26.080367 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 8c6202333af943cbbc0d09553c1c193c (table default.Table_fe566740b43c42658f0a0ff2de59038a [id=e9c7ec14573b44b79e70dc70fc9226c2]) successfully deleted
I20240503 14:12:26.080629 29828 raft_consensus.cc:2229] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.081185 29828 raft_consensus.cc:2258] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.081398 30142 tablet_replica.cc:330] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.081640 30142 raft_consensus.cc:2229] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.081970 29828 ts_tablet_manager.cc:1875] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.082028 30142 raft_consensus.cc:2258] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.078100 28604 raft_consensus.cc:374] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.085206 28604 raft_consensus.cc:729] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.085592 28604 consensus_queue.cc:260] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.087540 28604 ts_tablet_manager.cc:1398] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:12:26.088311 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9edfb0d064939bda2ec7a9e0e500b with delete_type TABLET_DATA_DELETED (replaced by tablet 1906ab252ea349cfa0afbd9f4303efbc) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.088735 30142 ts_tablet_manager.cc:1875] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.089020 28604 tablet_bootstrap.cc:492] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.090224 28614 tablet_bootstrap.cc:492] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:26.090396 28614 ts_tablet_manager.cc:1367] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.000s
I20240503 14:12:26.090910 28604 tablet_bootstrap.cc:654] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.091217 28614 raft_consensus.cc:348] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.091446 28614 raft_consensus.cc:374] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.091502 28614 raft_consensus.cc:729] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.091724 28614 consensus_queue.cc:260] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.092463 28614 ts_tablet_manager.cc:1398] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:26.092761 28614 tablet_bootstrap.cc:492] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:26.094416 28614 tablet_bootstrap.cc:654] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.096730 28600 tablet_bootstrap.cc:492] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:26.097031 28600 ts_tablet_manager.cc:1367] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.001s
I20240503 14:12:26.097923 28600 raft_consensus.cc:348] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.098222 28600 raft_consensus.cc:374] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.098351 28600 raft_consensus.cc:729] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.098709 28600 consensus_queue.cc:260] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.099440 28600 ts_tablet_manager.cc:1398] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:26.099812 28661 tablet_replica.cc:330] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:26.101799 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 77f0081cc2cb48aaa368741109032fd1 because the tablet was not found. No further retry: Not found: Tablet not found: 77f0081cc2cb48aaa368741109032fd1
I20240503 14:12:26.103403 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 495190fca06a40a386c8215eb2195920 (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]) successfully deleted
W20240503 14:12:26.103827 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7e1f3443c298426db9ff5df1437884f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1f3443c298426db9ff5df1437884f9 already in progress: creating tablet
I20240503 14:12:26.104043 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c77a11ef384709aef5b62273380573" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:26.104280 28214 raft_consensus.cc:3037] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:26.105183 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b0993be031cc4792a9eadba9ce41d024 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0993be031cc4792a9eadba9ce41d024 already in progress: creating tablet
W20240503 14:12:26.106421 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5087cf4a795f4ee08ce4dd19d095d0e5 already in progress: creating tablet
I20240503 14:12:26.106645 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e4a544fc27624dcc8a4cb003e59408f2 (table default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]) successfully deleted
W20240503 14:12:26.106868 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2946ba624b244db9a617f2e202bfc1ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2946ba624b244db9a617f2e202bfc1ec already in progress: creating tablet
W20240503 14:12:26.107060 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7b89711872ba442ca66e35091a2271da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b89711872ba442ca66e35091a2271da already in progress: creating tablet
I20240503 14:12:26.107061 28214 raft_consensus.cc:2454] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:26.113519 28600 tablet_bootstrap.cc:492] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.115427 28600 tablet_bootstrap.cc:654] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.103730 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c77a11ef384709aef5b62273380573" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:26.122952 28487 raft_consensus.cc:3037] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.125933 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0d4c5648ddc84407a8d8b44f2d63dcee
I20240503 14:12:26.125916 28487 raft_consensus.cc:2454] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:26.126904 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ddb29edceb57402988fa9a6aa01c4c1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:26.127355 28661 raft_consensus.cc:2229] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.127591 28661 raft_consensus.cc:2258] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.128219 29811 consensus_queue.cc:1035] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:26.128425 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddb29edceb57402988fa9a6aa01c4c1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.128527 28661 ts_tablet_manager.cc:1875] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.131630 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d4c5648ddc84407a8d8b44f2d63dcee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.132037 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d4c5648ddc84407a8d8b44f2d63dcee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.133196 29996 raft_consensus.cc:482] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.133365 29996 raft_consensus.cc:504] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.134025 29996 leader_election.cc:288] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:26.135370 28427 leader_election.cc:341] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: e4a544fc27624dcc8a4cb003e59408f2
I20240503 14:12:26.139524 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 19d43cc864064c4c907ec0629391da10
I20240503 14:12:26.140107 28030 catalog_manager.cc:6554] ReplaceTablet: tablet 19d43cc864064c4c907ec0629391da10 deleted and replaced by tablet d479f4f392244d5c877db4820367b224
I20240503 14:12:26.140632 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3d13d5e860f2457895d61f215a2c08cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.141287 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d13d5e860f2457895d61f215a2c08cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.142817 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 13f8553cfbab4af2a38a73e81fed2a5f
I20240503 14:12:26.143616 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_eb65ad8d07974a069b16d09a5dd36d1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.144245 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb65ad8d07974a069b16d09a5dd36d1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.146282 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9f5b973ea16d42babd2cc75e32624854
I20240503 14:12:26.146771 28025 catalog_manager.cc:6554] ReplaceTablet: tablet 9f5b973ea16d42babd2cc75e32624854 deleted and replaced by tablet 60ef46a8638248a9b9c46e48fbb8cfcc
I20240503 14:12:26.147027 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 77511682d71a40f5933edcb6a6523649 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:26.152590 28661 ts_tablet_manager.cc:1888] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.152740 28661 log.cc:1196] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/41cbd1326869434594e126421ad132d8
I20240503 14:12:26.153137 28661 ts_tablet_manager.cc:1909] T 41cbd1326869434594e126421ad132d8 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.153549 30129 consensus_queue.cc:1035] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.154016 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 891) took 3456 ms. Trace:
I20240503 14:12:26.154142 28661 rpcz_store.cc:276] 0503 14:12:22.697561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.697609 (+ 48us) service_pool.cc:225] Handling call
0503 14:12:26.099749 (+3402140us) ts_tablet_manager.cc:1120] Deleting tablet 41cbd1326869434594e126421ad132d8
0503 14:12:26.099761 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.137986 (+ 38225us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.152461 (+ 14475us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.153990 (+ 1529us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4197,"tablet-delete.queue_time_us":3314950}
I20240503 14:12:26.156704 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d43cc864064c4c907ec0629391da10 with delete_type TABLET_DATA_DELETED (replaced by tablet d479f4f392244d5c877db4820367b224) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.157510 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d43cc864064c4c907ec0629391da10 with delete_type TABLET_DATA_DELETED (replaced by tablet d479f4f392244d5c877db4820367b224) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.157618 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f8553cfbab4af2a38a73e81fed2a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.157685 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f8553cfbab4af2a38a73e81fed2a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.158319 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5b973ea16d42babd2cc75e32624854 with delete_type TABLET_DATA_DELETED (replaced by tablet 60ef46a8638248a9b9c46e48fbb8cfcc) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.159058 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 41cbd1326869434594e126421ad132d8 (table default.Table_a07e2f0650084270860bd28e40f224aa [id=a98b2a6e32bf425eba5d2e039def6b0f]) successfully deleted
I20240503 14:12:26.159091 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5b973ea16d42babd2cc75e32624854 with delete_type TABLET_DATA_DELETED (replaced by tablet 60ef46a8638248a9b9c46e48fbb8cfcc) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.161157 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd8fb8ea1a4fc3bfa4f55ea59ae12a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.161397 28354 raft_consensus.cc:2454] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:26.161973 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd8fb8ea1a4fc3bfa4f55ea59ae12a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:26.162163 28354 raft_consensus.cc:3037] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.164822 28354 raft_consensus.cc:2454] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.167347 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 77c77a11ef384709aef5b62273380573 (table default.Table_ab495b10ad82425ba5c7123a6e1cf064 [id=2edd1164c07a4a67b51262d402d53596]). Expected version 2 got 0
I20240503 14:12:26.167922 28024 catalog_manager.cc:5484] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:26.168453 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet b75036cdf9e04ea4aa767fd12016f177 (table default.Table_87bcc8d1e5244ebdb6ed58c62b34f709 [id=30ff854aabe84b2cb03824d287ae5685]). Expected version 1 got 0
I20240503 14:12:26.169159 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:26.170051 28031 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_881b9fb0c89441bcb58040654dc3db10 [id=874fc82c70674371a8be82614bb6fcfb] alter complete (version 6)
I20240503 14:12:26.172564 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_edb926a0a57b4cb090017b29fee48eab" } new_table_name: "default.Table_dc05b86aa4894b12850852f8f2344659" modify_external_catalogs: true
W20240503 14:12:26.175837 27856 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 82395.6us
I20240503 14:12:26.176697 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 892) took 3468 ms. Trace:
I20240503 14:12:26.176810 28661 rpcz_store.cc:276] 0503 14:12:22.708097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.708160 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:26.173164 (+3465004us) ts_tablet_manager.cc:1120] Deleting tablet b04fff656f624001a709f783816eec1e
0503 14:12:26.176518 (+ 3354us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.176667 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3388312}
I20240503 14:12:26.178455 29996 raft_consensus.cc:482] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.178611 29996 raft_consensus.cc:504] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.179314 29996 leader_election.cc:288] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.180158 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:26.180495 28214 raft_consensus.cc:2454] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:26.181746 28427 leader_election.cc:302] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.185262 30142 ts_tablet_manager.cc:1888] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.185420 30134 raft_consensus.cc:2781] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:26.185523 30134 raft_consensus.cc:482] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:26.185592 30134 raft_consensus.cc:3037] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.187611 30142 log.cc:1196] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/80cd09e769be4bee95bbef9c3c550451
I20240503 14:12:26.189278 30134 raft_consensus.cc:504] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.189851 30142 ts_tablet_manager.cc:1909] T 80cd09e769be4bee95bbef9c3c550451 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.190050 30134 leader_election.cc:288] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.190632 30107 raft_consensus.cc:482] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.178604 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 911) took 3149 ms. Trace:
I20240503 14:12:26.190793 28661 rpcz_store.cc:276] 0503 14:12:23.029506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.029553 (+ 47us) service_pool.cc:225] Handling call
0503 14:12:26.177321 (+3147768us) ts_tablet_manager.cc:1120] Deleting tablet 0c300b1f104949b3ae74f5b38d378a11
0503 14:12:26.178454 (+ 1133us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.178588 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3147544}
I20240503 14:12:26.190815 30107 raft_consensus.cc:504] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.191388 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 912) took 3161 ms. Trace:
I20240503 14:12:26.191452 30107 leader_election.cc:288] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:26.191465 28661 rpcz_store.cc:276] 0503 14:12:23.029980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.030022 (+ 42us) service_pool.cc:225] Handling call
0503 14:12:26.191241 (+3161219us) ts_tablet_manager.cc:1120] Deleting tablet 86804987bdf340dc83ddd58ba2bc8a96
0503 14:12:26.191252 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.191367 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3161058}
I20240503 14:12:26.192050 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 909) took 3162 ms. Trace:
I20240503 14:12:26.192147 28661 rpcz_store.cc:276] 0503 14:12:23.029348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.036316 (+ 6968us) service_pool.cc:225] Handling call
0503 14:12:26.191927 (+3155611us) ts_tablet_manager.cc:1120] Deleting tablet 00a56d79c36f4ffa9bae02e2369d8ded
0503 14:12:26.191937 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.192033 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3155263}
I20240503 14:12:26.192109 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:26.192695 28212 raft_consensus.cc:3037] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.194500 29828 ts_tablet_manager.cc:1888] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:26.194619 29828 log.cc:1196] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/d93fea5caec94695a5df26d3630c50c7
I20240503 14:12:26.194887 29828 ts_tablet_manager.cc:1909] T d93fea5caec94695a5df26d3630c50c7 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:26.195379 29828 tablet_replica.cc:330] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:26.195377 28212 raft_consensus.cc:2454] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.195432 28661 tablet_replica.cc:330] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.195586 29828 raft_consensus.cc:2229] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.195777 29828 pending_rounds.cc:62] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:26.195941 28427 leader_election.cc:302] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.196283 30134 raft_consensus.cc:2781] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.196413 30134 raft_consensus.cc:686] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:26.196662 30134 consensus_queue.cc:237] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.206286 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.207055 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.207731 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.208355 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.209260 28661 raft_consensus.cc:2229] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.209681 28661 pending_rounds.cc:62] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:26.209743 28661 pending_rounds.cc:68] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597744135061504 op_type: NO_OP noop_request { }
I20240503 14:12:26.209865 28661 raft_consensus.cc:2866] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:26.209957 28661 raft_consensus.cc:2258] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.209964 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6202333af943cbbc0d09553c1c193c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.211036 28600 tablet_bootstrap.cc:492] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:26.211103 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 01087bdc04e74d189045de829379d6cd (DEFAULT_TABLE table=default.Table_4237d991665c4015b7d01b31d99f2370 [id=0a904a0fecd84b79898bc29e386179d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.211278 28600 ts_tablet_manager.cc:1367] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.099s user 0.014s sys 0.000s
I20240503 14:12:26.212206 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01087bdc04e74d189045de829379d6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.212986 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 34e1de683f344609a52aee90608353b1 (DEFAULT_TABLE table=default.Table_8d3a69435d7b492c88dd63b58b055c6d [id=0997e26c4a31483fb9b0174c7c47c7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.213423 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e1de683f344609a52aee90608353b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.214306 28600 raft_consensus.cc:348] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.214540 28600 raft_consensus.cc:374] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.214610 28600 raft_consensus.cc:729] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.214850 28600 consensus_queue.cc:260] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.215752 28600 ts_tablet_manager.cc:1398] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:26.219233 28600 tablet_bootstrap.cc:492] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.219380 30142 tablet_replica.cc:330] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.219669 30142 raft_consensus.cc:2229] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.220675 30142 raft_consensus.cc:2258] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.220759 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 8da2c8f097dd48acb84d42f3e3a3078e (DEFAULT_TABLE table=default.Table_f6b29fc1fc8b4818a92c622169ec4bd5 [id=9b55d096a9c04b1396aba081ffee517d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.222843 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:26.223696 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 70ec7e7462a54e51a86286a04a9e62ea (DEFAULT_TABLE table=default.Table_d62c11b873954239b3c4c78e716c2144 [id=64f3b1eedaf846439a6664c2e6faac77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.224402 30142 ts_tablet_manager.cc:1875] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.224653 28604 tablet_bootstrap.cc:492] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.224804 28604 ts_tablet_manager.cc:1367] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.001s
I20240503 14:12:26.225610 28604 raft_consensus.cc:348] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.225803 28604 raft_consensus.cc:374] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.225858 28604 raft_consensus.cc:729] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.226063 28604 consensus_queue.cc:260] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.226377 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8da2c8f097dd48acb84d42f3e3a3078e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.226639 28604 ts_tablet_manager.cc:1398] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:26.226616 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:26.226878 28213 raft_consensus.cc:2379] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:26.226917 28604 tablet_bootstrap.cc:492] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.228370 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ec7e7462a54e51a86286a04a9e62ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.231392 28604 tablet_bootstrap.cc:654] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.234793 28600 tablet_bootstrap.cc:654] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.235354 28661 ts_tablet_manager.cc:1875] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.237361 30142 ts_tablet_manager.cc:1888] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.237597 30142 log.cc:1196] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ff4d28d3c2ea472191ad21355248a2fc
I20240503 14:12:26.237567 30134 raft_consensus.cc:482] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.237699 30134 raft_consensus.cc:504] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.238181 30142 ts_tablet_manager.cc:1909] T ff4d28d3c2ea472191ad21355248a2fc P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.238322 30134 leader_election.cc:288] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.239372 30142 tablet_replica.cc:330] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.240056 30142 raft_consensus.cc:2229] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.240645 30142 raft_consensus.cc:2258] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.241757 30142 ts_tablet_manager.cc:1875] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.253319 28180 tablet_service.cc:1467] Processing CreateTablet for tablet d2eb8328e86a4ee5a53647a810729519 (DEFAULT_TABLE table=default.Table_0573272ac5c84afc82be96eea79f4ce3 [id=e65a786348b74b268e37fc75e50922d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.253788 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2eb8328e86a4ee5a53647a810729519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.255646 30142 ts_tablet_manager.cc:1888] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.255777 30142 log.cc:1196] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2221218847554cbd9d3242a9a6431e46
I20240503 14:12:26.259166 30142 ts_tablet_manager.cc:1909] T 2221218847554cbd9d3242a9a6431e46 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.263653 30142 tablet_replica.cc:330] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.263960 30142 raft_consensus.cc:2229] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.264246 30142 raft_consensus.cc:2258] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.264379 28600 tablet_bootstrap.cc:492] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:26.264511 28600 ts_tablet_manager.cc:1367] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.001s
I20240503 14:12:26.265368 30142 ts_tablet_manager.cc:1875] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.265313 28600 raft_consensus.cc:348] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.265516 28600 raft_consensus.cc:374] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.265568 28600 raft_consensus.cc:729] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.265834 28600 consensus_queue.cc:260] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.266362 28600 ts_tablet_manager.cc:1398] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:26.266640 28600 tablet_bootstrap.cc:492] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.267975 28600 tablet_bootstrap.cc:654] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.274866 28661 ts_tablet_manager.cc:1888] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.277954 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 80cd09e769be4bee95bbef9c3c550451 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
W20240503 14:12:26.278438 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ed4dc732bcb44222b4b15e5d77defe33 because the tablet was not found. No further retry: Not found: Tablet not found: ed4dc732bcb44222b4b15e5d77defe33
I20240503 14:12:26.278764 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ff4d28d3c2ea472191ad21355248a2fc (table default.Table_7b9e454770964e4594e5d043b06c2e78 [id=d4a1074afcca45b287b8d4416878f316]) successfully deleted
I20240503 14:12:26.278877 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 0311cfbb060e439d9b7c497ffe4626cd (DEFAULT_TABLE table=default.Table_53c1790884a24325abc8160a18f4d9cf [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.278980 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2221218847554cbd9d3242a9a6431e46 (table default.Table_4929b450373c4b689562f4a1d9e4b4f2 [id=7b85025507014faa9ce25ba44310b973]) successfully deleted
I20240503 14:12:26.279407 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0311cfbb060e439d9b7c497ffe4626cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:26.279376 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b04fff656f624001a709f783816eec1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b04fff656f624001a709f783816eec1e already in progress: creating tablet
W20240503 14:12:26.279613 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0c300b1f104949b3ae74f5b38d378a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c300b1f104949b3ae74f5b38d378a11 already in progress: creating tablet
W20240503 14:12:26.279819 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 86804987bdf340dc83ddd58ba2bc8a96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86804987bdf340dc83ddd58ba2bc8a96 already in progress: creating tablet
I20240503 14:12:26.281775 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 773bc744579b4ff28c950ea632753221 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.282171 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773bc744579b4ff28c950ea632753221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.284198 28661 log.cc:1196] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ed4dc732bcb44222b4b15e5d77defe33
I20240503 14:12:26.284713 28661 ts_tablet_manager.cc:1909] T ed4dc732bcb44222b4b15e5d77defe33 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:26.293126 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 63952.6us
I20240503 14:12:26.293383 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 913) took 3243 ms. Trace:
W20240503 14:12:26.293975 28427 leader_election.cc:341] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: e4a544fc27624dcc8a4cb003e59408f2
I20240503 14:12:26.294152 28427 leader_election.cc:302] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:26.293975 28661 rpcz_store.cc:276] 0503 14:12:23.050131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.066518 (+ 16387us) service_pool.cc:225] Handling call
0503 14:12:26.195381 (+3128863us) ts_tablet_manager.cc:1120] Deleting tablet ed4dc732bcb44222b4b15e5d77defe33
0503 14:12:26.195393 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.247448 (+ 52055us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.274731 (+ 27283us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.293338 (+ 18607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3128423}
I20240503 14:12:26.294458 29994 raft_consensus.cc:2726] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.295080 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 910) took 3265 ms. Trace:
I20240503 14:12:26.295326 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 01087bdc04e74d189045de829379d6cd (DEFAULT_TABLE table=default.Table_4237d991665c4015b7d01b31d99f2370 [id=0a904a0fecd84b79898bc29e386179d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.295414 28661 rpcz_store.cc:276] 0503 14:12:23.029436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.067264 (+ 37828us) service_pool.cc:225] Handling call
0503 14:12:26.294916 (+3227652us) ts_tablet_manager.cc:1120] Deleting tablet 380f267a984541c7878eeaa4eeee8f34
0503 14:12:26.294928 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.295032 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227373}
I20240503 14:12:26.295625 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 34e1de683f344609a52aee90608353b1 (DEFAULT_TABLE table=default.Table_8d3a69435d7b492c88dd63b58b055c6d [id=0997e26c4a31483fb9b0174c7c47c7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.295794 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01087bdc04e74d189045de829379d6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.296061 28459 tablet_service.cc:1467] Processing CreateTablet for tablet d2eb8328e86a4ee5a53647a810729519 (DEFAULT_TABLE table=default.Table_0573272ac5c84afc82be96eea79f4ce3 [id=e65a786348b74b268e37fc75e50922d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.295624 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 70ec7e7462a54e51a86286a04a9e62ea (DEFAULT_TABLE table=default.Table_d62c11b873954239b3c4c78e716c2144 [id=64f3b1eedaf846439a6664c2e6faac77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.295804 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 922) took 2307 ms. Trace:
I20240503 14:12:26.297080 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b176a071ba4f3fbff2a64d6e46027a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:26.297305 28215 raft_consensus.cc:2454] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:26.297315 29994 raft_consensus.cc:482] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.297467 29994 raft_consensus.cc:504] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.297966 28600 tablet_bootstrap.cc:492] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:26.298091 29994 leader_election.cc:288] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:26.298195 28600 ts_tablet_manager.cc:1367] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.003s
I20240503 14:12:26.297171 28661 rpcz_store.cc:276] 0503 14:12:23.988146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.988203 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:26.295658 (+2307455us) ts_tablet_manager.cc:1120] Deleting tablet 2b9a6aca21d746eb8f295ed5ea44c9d9
0503 14:12:26.295668 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.295752 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2307296}
I20240503 14:12:26.296674 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 0311cfbb060e439d9b7c497ffe4626cd (DEFAULT_TABLE table=default.Table_53c1790884a24325abc8160a18f4d9cf [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.296736 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 8da2c8f097dd48acb84d42f3e3a3078e (DEFAULT_TABLE table=default.Table_f6b29fc1fc8b4818a92c622169ec4bd5 [id=9b55d096a9c04b1396aba081ffee517d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.296857 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e1de683f344609a52aee90608353b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.296337 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 773bc744579b4ff28c950ea632753221 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.299559 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2eb8328e86a4ee5a53647a810729519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.298882 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 918) took 2311 ms. Trace:
I20240503 14:12:26.299890 28661 rpcz_store.cc:276] 0503 14:12:23.987129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.987207 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:26.298472 (+2311265us) ts_tablet_manager.cc:1120] Deleting tablet 1132014802ed4c3682769ba86312eed2
0503 14:12:26.298715 (+ 243us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.298839 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2246633}
I20240503 14:12:26.300148 29828 pending_rounds.cc:68] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597749871136768 op_type: NO_OP noop_request { }
I20240503 14:12:26.300287 29828 raft_consensus.cc:2866] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:26.300379 29828 raft_consensus.cc:2258] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.300969 29828 ts_tablet_manager.cc:1875] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.209117 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.303095 28350 raft_consensus.cc:2454] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:26.209280 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8411f474c247c2abfd0bd9682fb8a2" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:26.252161 28614 tablet_bootstrap.cc:492] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:26.288656 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d43cc864064c4c907ec0629391da10 with delete_type TABLET_DATA_DELETED (replaced by tablet d479f4f392244d5c877db4820367b224) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.288947 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f8553cfbab4af2a38a73e81fed2a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.303380 28357 raft_consensus.cc:3037] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.289189 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5b973ea16d42babd2cc75e32624854 with delete_type TABLET_DATA_DELETED (replaced by tablet 60ef46a8638248a9b9c46e48fbb8cfcc) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.289652 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 01087bdc04e74d189045de829379d6cd (DEFAULT_TABLE table=default.Table_4237d991665c4015b7d01b31d99f2370 [id=0a904a0fecd84b79898bc29e386179d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.303763 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01087bdc04e74d189045de829379d6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.304061 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ec7e7462a54e51a86286a04a9e62ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.306044 28357 raft_consensus.cc:2454] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.309082 28600 raft_consensus.cc:348] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.309402 28600 raft_consensus.cc:374] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.289980 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d4c5648ddc84407a8d8b44f2d63dcee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.290241 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 34e1de683f344609a52aee90608353b1 (DEFAULT_TABLE table=default.Table_8d3a69435d7b492c88dd63b58b055c6d [id=0997e26c4a31483fb9b0174c7c47c7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.309497 28600 raft_consensus.cc:729] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.309820 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e1de683f344609a52aee90608353b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.309777 28600 consensus_queue.cc:260] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.290694 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 8da2c8f097dd48acb84d42f3e3a3078e (DEFAULT_TABLE table=default.Table_f6b29fc1fc8b4818a92c622169ec4bd5 [id=9b55d096a9c04b1396aba081ffee517d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.312404 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8da2c8f097dd48acb84d42f3e3a3078e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.314257 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0311cfbb060e439d9b7c497ffe4626cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.314880 28600 ts_tablet_manager.cc:1398] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240503 14:12:26.291105 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 70ec7e7462a54e51a86286a04a9e62ea (DEFAULT_TABLE table=default.Table_d62c11b873954239b3c4c78e716c2144 [id=64f3b1eedaf846439a6664c2e6faac77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.316370 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8da2c8f097dd48acb84d42f3e3a3078e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.316411 28600 tablet_bootstrap.cc:492] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.316591 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ec7e7462a54e51a86286a04a9e62ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.291663 28320 tablet_service.cc:1467] Processing CreateTablet for tablet d2eb8328e86a4ee5a53647a810729519 (DEFAULT_TABLE table=default.Table_0573272ac5c84afc82be96eea79f4ce3 [id=e65a786348b74b268e37fc75e50922d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.320469 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2eb8328e86a4ee5a53647a810729519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.322067 28600 tablet_bootstrap.cc:654] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.324510 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773bc744579b4ff28c950ea632753221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.324520 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 921) took 2336 ms. Trace:
I20240503 14:12:26.324690 28661 rpcz_store.cc:276] 0503 14:12:23.987932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.987975 (+ 43us) service_pool.cc:225] Handling call
0503 14:12:26.302941 (+2314966us) ts_tablet_manager.cc:1120] Deleting tablet 11e14e8b5f4046029af4673a277112d9
0503 14:12:26.324308 (+ 21367us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.324455 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235759}
I20240503 14:12:26.325081 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 924) took 2335 ms. Trace:
I20240503 14:12:26.325182 28661 rpcz_store.cc:276] 0503 14:12:23.989874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.989923 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:26.324955 (+2335032us) ts_tablet_manager.cc:1120] Deleting tablet d21bdd6490004300bca7a5a7fb24cd07
0503 14:12:26.324965 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.325044 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2257629}
I20240503 14:12:26.325527 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 940) took 1897 ms. Trace:
I20240503 14:12:26.325634 28661 rpcz_store.cc:276] 0503 14:12:24.428013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.449192 (+ 21179us) service_pool.cc:225] Handling call
0503 14:12:26.325406 (+1876214us) ts_tablet_manager.cc:1120] Deleting tablet abb9279e91124fe6bfbdc8856eccfed8
0503 14:12:26.325415 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.325487 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1865129}
I20240503 14:12:26.326249 28661 tablet_replica.cc:330] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.326478 28661 raft_consensus.cc:2229] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.326650 28661 raft_consensus.cc:2258] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.327278 28661 ts_tablet_manager.cc:1875] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.327570 28614 ts_tablet_manager.cc:1367] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.235s user 0.002s sys 0.001s
I20240503 14:12:26.291728 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 773bc744579b4ff28c950ea632753221 (DEFAULT_TABLE table=default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.328421 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773bc744579b4ff28c950ea632753221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.329255 28614 raft_consensus.cc:348] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.329491 28614 raft_consensus.cc:374] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.329566 28614 raft_consensus.cc:729] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.329808 28614 consensus_queue.cc:260] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.330480 28614 ts_tablet_manager.cc:1398] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:26.330794 28614 tablet_bootstrap.cc:492] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:26.292299 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 0311cfbb060e439d9b7c497ffe4626cd (DEFAULT_TABLE table=default.Table_53c1790884a24325abc8160a18f4d9cf [id=87c5214d069a46dd936e08642c0c923c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.331285 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0311cfbb060e439d9b7c497ffe4626cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.292496 28290 leader_election.cc:302] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.331859 30107 raft_consensus.cc:2781] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.331993 30107 raft_consensus.cc:686] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:26.332264 30107 consensus_queue.cc:237] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.333964 28661 ts_tablet_manager.cc:1888] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:26.334147 28661 log.cc:1196] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e4a544fc27624dcc8a4cb003e59408f2
I20240503 14:12:26.336547 28661 ts_tablet_manager.cc:1909] T e4a544fc27624dcc8a4cb003e59408f2 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.338985 28604 tablet_bootstrap.cc:492] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.341626 28604 ts_tablet_manager.cc:1367] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.000s
I20240503 14:12:26.342366 28661 tablet_replica.cc:330] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.344297 28661 raft_consensus.cc:2229] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.346834 28604 raft_consensus.cc:348] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:26.347280 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 54780.7us
I20240503 14:12:26.347677 28604 raft_consensus.cc:374] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.350106 28604 raft_consensus.cc:729] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.350446 28604 consensus_queue.cc:260] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.351279 28604 ts_tablet_manager.cc:1398] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:12:26.351645 28604 tablet_bootstrap.cc:492] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.348477 28661 raft_consensus.cc:2258] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.353508 28604 tablet_bootstrap.cc:654] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.355381 28661 ts_tablet_manager.cc:1875] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:26.280040 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 00a56d79c36f4ffa9bae02e2369d8ded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00a56d79c36f4ffa9bae02e2369d8ded already in progress: creating tablet
W20240503 14:12:26.364336 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 84291.3us
I20240503 14:12:26.369267 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet addc2a6195034b0a927fdee7a2955122
I20240503 14:12:26.370170 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d94a92f094354a9abb3e0f0173679568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.370873 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d94a92f094354a9abb3e0f0173679568 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.376694 28614 tablet_bootstrap.cc:654] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.382424 30142 ts_tablet_manager.cc:1888] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.383319 30142 log.cc:1196] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/4084bd4fbf604042bb33ff03c3f8cc19
I20240503 14:12:26.383883 30142 ts_tablet_manager.cc:1909] T 4084bd4fbf604042bb33ff03c3f8cc19 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.385085 28427 leader_election.cc:302] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.385394 29994 raft_consensus.cc:2781] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:12:26.385424 28427 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:26.385489 29994 raft_consensus.cc:482] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:12:26.386015 28427 consensus_peers.cc:485] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c6202333af943cbbc0d09553c1c193c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:26.387109 28290 leader_election.cc:302] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.387171 29994 raft_consensus.cc:3037] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.387645 30107 raft_consensus.cc:2726] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.389822 28661 ts_tablet_manager.cc:1888] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.389755 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet addc2a6195034b0a927fdee7a2955122 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.390625 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet d93fea5caec94695a5df26d3630c50c7 (table default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]) successfully deleted
I20240503 14:12:26.391180 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ed4dc732bcb44222b4b15e5d77defe33 (table default.Table_3ce7004b7c164629a92e6bcbb421348d [id=a6a9ad0f171448c2a3d3401e02ec62c0]) successfully deleted
W20240503 14:12:26.391477 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 380f267a984541c7878eeaa4eeee8f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 380f267a984541c7878eeaa4eeee8f34 already in progress: creating tablet
W20240503 14:12:26.391716 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9a6aca21d746eb8f295ed5ea44c9d9 already in progress: creating tablet
W20240503 14:12:26.391950 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 1132014802ed4c3682769ba86312eed2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1132014802ed4c3682769ba86312eed2 already in progress: creating tablet
I20240503 14:12:26.392252 30142 tablet_replica.cc:330] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:26.392316 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 11e14e8b5f4046029af4673a277112d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11e14e8b5f4046029af4673a277112d9 already in progress: creating tablet
W20240503 14:12:26.392570 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d21bdd6490004300bca7a5a7fb24cd07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d21bdd6490004300bca7a5a7fb24cd07 already in progress: creating tablet
W20240503 14:12:26.392783 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet abb9279e91124fe6bfbdc8856eccfed8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abb9279e91124fe6bfbdc8856eccfed8 already in progress: creating tablet
W20240503 14:12:26.393007 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b0993be031cc4792a9eadba9ce41d024 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0993be031cc4792a9eadba9ce41d024 already in progress: creating tablet
W20240503 14:12:26.393222 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7e1f3443c298426db9ff5df1437884f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1f3443c298426db9ff5df1437884f9 already in progress: creating tablet
I20240503 14:12:26.393352 30142 raft_consensus.cc:2229] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:26.393461 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5087cf4a795f4ee08ce4dd19d095d0e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5087cf4a795f4ee08ce4dd19d095d0e5 already in progress: creating tablet
I20240503 14:12:26.393638 30142 raft_consensus.cc:2258] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.393649 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e4a544fc27624dcc8a4cb003e59408f2 (table default.Table_afe079d9fefc4dd3af481024b84ce6a2 [id=575a7bcbb4e34b17b09fd7b05cebbd65]) successfully deleted
W20240503 14:12:26.393884 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2946ba624b244db9a617f2e202bfc1ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2946ba624b244db9a617f2e202bfc1ec already in progress: creating tablet
W20240503 14:12:26.394069 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7b89711872ba442ca66e35091a2271da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b89711872ba442ca66e35091a2271da already in progress: creating tablet
W20240503 14:12:26.394290 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 77f0081cc2cb48aaa368741109032fd1 because the tablet was not found. No further retry: Not found: Tablet not found: 77f0081cc2cb48aaa368741109032fd1
I20240503 14:12:26.394876 30142 ts_tablet_manager.cc:1875] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.389890 29994 raft_consensus.cc:504] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.396283 29994 leader_election.cc:288] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.396615 28661 log.cc:1196] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/2221218847554cbd9d3242a9a6431e46
I20240503 14:12:26.398948 28661 ts_tablet_manager.cc:1909] T 2221218847554cbd9d3242a9a6431e46 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.399546 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b176a071ba4f3fbff2a64d6e46027a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.399785 28357 raft_consensus.cc:2454] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:26.400952 28661 tablet_replica.cc:330] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.406291 28661 raft_consensus.cc:2229] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.407008 28661 raft_consensus.cc:2258] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.409577 28661 ts_tablet_manager.cc:1875] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.410800 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet addc2a6195034b0a927fdee7a2955122 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.404946 30005 raft_consensus.cc:482] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.413316 30005 raft_consensus.cc:504] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.414000 30005 leader_election.cc:288] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.414584 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50b3e199314bfc8465fdb7b2425d07" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:26.415205 28150 leader_election.cc:341] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.416185 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b176a071ba4f3fbff2a64d6e46027a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:26.416395 28215 raft_consensus.cc:3037] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.419016 28215 raft_consensus.cc:2454] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.414553 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50b3e199314bfc8465fdb7b2425d07" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:26.420614 28486 raft_consensus.cc:2454] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:26.421100 28150 leader_election.cc:302] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:26.421447 30005 raft_consensus.cc:2781] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:26.421536 30005 raft_consensus.cc:482] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:26.421602 30005 raft_consensus.cc:3037] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.424692 28604 tablet_bootstrap.cc:492] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.424841 28604 ts_tablet_manager.cc:1367] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20240503 14:12:26.425709 28427 leader_election.cc:302] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.425981 29994 raft_consensus.cc:2781] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.426738 30005 raft_consensus.cc:504] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.429633 29994 raft_consensus.cc:686] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:26.429960 29994 consensus_queue.cc:237] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.430981 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 4084bd4fbf604042bb33ff03c3f8cc19 (table default.Table_3a0134977dde48a0849c29c36487c742 [id=119495cd0b64414690f0d19bcce2ed78]) successfully deleted
I20240503 14:12:26.431352 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 2221218847554cbd9d3242a9a6431e46 (table default.Table_4929b450373c4b689562f4a1d9e4b4f2 [id=7b85025507014faa9ce25ba44310b973]) successfully deleted
W20240503 14:12:26.431581 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e0a92fc3833e49aaab8c5415de9bb685 because the tablet was not found. No further retry: Not found: Tablet not found: e0a92fc3833e49aaab8c5415de9bb685
I20240503 14:12:26.431517 29811 raft_consensus.cc:482] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.433007 29811 raft_consensus.cc:504] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.433697 29811 leader_election.cc:288] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.432585 30129 consensus_queue.cc:1035] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.435942 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884cf4ea11e48ef941722b79ad50075" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:26.436759 28150 leader_election.cc:341] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.437232 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet addc2a6195034b0a927fdee7a2955122 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.447607 29811 raft_consensus.cc:482] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.447762 29811 raft_consensus.cc:504] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.448412 29811 leader_election.cc:288] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.448971 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9325d5142af4b018354bf38370b61e0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:26.449443 30005 leader_election.cc:288] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.450474 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50b3e199314bfc8465fdb7b2425d07" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:26.450668 28493 raft_consensus.cc:3037] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.453436 28493 raft_consensus.cc:2454] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
W20240503 14:12:26.454437 28150 leader_election.cc:341] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.454685 28150 leader_election.cc:302] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.454973 30005 raft_consensus.cc:2781] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.455106 30005 raft_consensus.cc:686] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:26.455400 30005 consensus_queue.cc:237] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
W20240503 14:12:26.460189 28427 leader_election.cc:341] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Not found: Tablet not found: 0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:26.460723 28427 leader_election.cc:302] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:26.462111 28600 tablet_bootstrap.cc:492] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:26.462347 28600 ts_tablet_manager.cc:1367] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.146s user 0.006s sys 0.022s
I20240503 14:12:26.463142 28600 raft_consensus.cc:348] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.463449 28600 raft_consensus.cc:374] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.463544 28600 raft_consensus.cc:729] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.463783 28600 consensus_queue.cc:260] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:26.464357 28600 ts_tablet_manager.cc:1398] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:26.464345 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 77511682d71a40f5933edcb6a6523649
I20240503 14:12:26.464677 28600 tablet_bootstrap.cc:492] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.464821 28025 catalog_manager.cc:6554] ReplaceTablet: tablet 77511682d71a40f5933edcb6a6523649 deleted and replaced by tablet a88cf5de223f4a6abc21691fea1c4da0
W20240503 14:12:26.465842 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3150dfcd21bc4604be8f9c2cbdc10648 consensus state has no leader
I20240503 14:12:26.469038 28600 tablet_bootstrap.cc:654] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.471185 28614 tablet_bootstrap.cc:492] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:26.471185 29828 ts_tablet_manager.cc:1888] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.473277 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b176a071ba4f3fbff2a64d6e46027a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:26.477785 30142 ts_tablet_manager.cc:1888] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:26.478992 28614 ts_tablet_manager.cc:1367] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.000s
I20240503 14:12:26.480341 30142 log.cc:1196] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/8fe3affd0d144b1fb0078e7690b9b36d
I20240503 14:12:26.480942 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884cf4ea11e48ef941722b79ad50075" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.481143 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 77511682d71a40f5933edcb6a6523649 with delete_type TABLET_DATA_DELETED (replaced by tablet a88cf5de223f4a6abc21691fea1c4da0) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.481943 29994 raft_consensus.cc:2726] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.483186 28614 raft_consensus.cc:348] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.484040 28614 raft_consensus.cc:374] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.484119 28350 raft_consensus.cc:3037] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.479326 29828 log.cc:1196] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ae1cd28bc07e4ccd9097f6dee471c5c6
W20240503 14:12:26.486430 28150 leader_election.cc:341] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.486563 28150 leader_election.cc:302] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.488597 29811 raft_consensus.cc:2726] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.488804 29811 consensus_queue.cc:1035] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.489753 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 77511682d71a40f5933edcb6a6523649 with delete_type TABLET_DATA_DELETED (replaced by tablet a88cf5de223f4a6abc21691fea1c4da0) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.492501 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet c99c15b4d13647e98cae1375b7f13827
I20240503 14:12:26.497781 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:26.486088 30070 consensus_queue.cc:1035] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.498100 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet b75036cdf9e04ea4aa767fd12016f177 (table default.Table_87bcc8d1e5244ebdb6ed58c62b34f709 [id=30ff854aabe84b2cb03824d287ae5685]). Expected version 1 got 0
I20240503 14:12:26.498467 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 77c77a11ef384709aef5b62273380573 (table default.Table_ab495b10ad82425ba5c7123a6e1cf064 [id=2edd1164c07a4a67b51262d402d53596]). Expected version 2 got 0
I20240503 14:12:26.473181 28604 raft_consensus.cc:348] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.460290 28661 ts_tablet_manager.cc:1888] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.501507 28604 raft_consensus.cc:374] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.501569 28661 log.cc:1196] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ff4d28d3c2ea472191ad21355248a2fc
I20240503 14:12:26.501645 28604 raft_consensus.cc:729] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.501523 28033 catalog_manager.cc:5484] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193). New cstate: current_term: 1 leader_uuid: "a64bc15d65d84ad5a005960a9a84df9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:26.502032 28661 ts_tablet_manager.cc:1909] T ff4d28d3c2ea472191ad21355248a2fc P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.501924 28604 consensus_queue.cc:260] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:26.506381 28024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 88) took 764 ms (client timeout 1000 ms). Trace:
W20240503 14:12:26.506633 28024 rpcz_store.cc:269] 0503 14:12:25.741886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.945540 (+203654us) service_pool.cc:225] Handling call
0503 14:12:26.002644 (+ 57104us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:26.506315 (+503671us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:26.179906 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:26.180015 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:26.180021 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:26.180023 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:26.180187 (+ 164us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:26.180198 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:26.180203 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:26.180268 (+ 65us) tablet.cc:699] Row locks acquired
0503 14:12:26.180271 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:26.180312 (+ 41us) write_op.cc:270] Start()
0503 14:12:26.180358 (+ 46us) write_op.cc:276] Timestamp: P: 1714745546180277 usec, L: 0
0503 14:12:26.180362 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:26.180547 (+ 185us) log.cc:842] Serialized 533 byte log entry
0503 14:12:26.490999 (+310452us) op_driver.cc:444] REPLICATION: finished
0503 14:12:26.491139 (+ 140us) write_op.cc:301] APPLY: starting
0503 14:12:26.491156 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:26.491195 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:26.491221 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:26.491295 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:26.491341 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:26.491348 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:26.491428 (+ 80us) log.cc:842] Serialized 23 byte log entry
0503 14:12:26.491490 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:26.491508 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:26.491516 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:26.491570 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:26.491573 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"spinlock_wait_cycles":129245952,"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":486,"num_ops":1,"prepare.queue_time_us":5434,"prepare.run_cpu_time_us":747,"prepare.run_wall_time_us":1151,"raft.queue_time_us":74,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":310612}]]}
I20240503 14:12:26.509029 30142 ts_tablet_manager.cc:1909] T 8fe3affd0d144b1fb0078e7690b9b36d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:26.513865 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 99aa7e59c45c45c7bc1f726c97426b89 (table default.Table_a5de24e133584200af645a728b1346e1 [id=8894157a06784f4288235959f27849b8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 99aa7e59c45c45c7bc1f726c97426b89 consensus state has no leader
W20240503 14:12:26.515009 28026 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 107) took 773 ms (client timeout 1000 ms). Trace:
W20240503 14:12:26.515158 28026 rpcz_store.cc:269] 0503 14:12:25.741430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.943646 (+202216us) service_pool.cc:225] Handling call
0503 14:12:26.004635 (+ 60989us) spinlock_profiling.cc:243] Waited 60 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:26.514961 (+510326us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":700,"reported_tablets":15,"spinlock_wait_cycles":138093312}
I20240503 14:12:26.517007 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b7ca19dd15f48a68ace7fefc7ae660f from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:26.518395 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d2eb8328e86a4ee5a53647a810729519
I20240503 14:12:26.513427 30142 tablet_replica.cc:330] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.518751 30142 raft_consensus.cc:2229] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.519110 30142 raft_consensus.cc:2258] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.519716 30142 ts_tablet_manager.cc:1875] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.523227 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2cb9f4b3da44486fbb7fdbf277833634" } new_table_name: "default.Table_c610fba592744fe294d3cdcee237091c" modify_external_catalogs: true
I20240503 14:12:26.524348 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_091731e3847544b1b411f7fe14d3182a" } new_table_name: "default.Table_f03c0efdc4da4e79831daffc520f9c00" modify_external_catalogs: true
I20240503 14:12:26.525897 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b15ac98442fe4481b0495a342ae6a57a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.526528 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b15ac98442fe4481b0495a342ae6a57a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.527992 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4fb75e6e800e495c9dfda4864bdebe5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.528590 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fb75e6e800e495c9dfda4864bdebe5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.532260 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1f00f0c6b19348a0a19c3045224268ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.532881 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f00f0c6b19348a0a19c3045224268ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.534406 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_70505af08e3d4b4aa9b097f33cae154a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.534950 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70505af08e3d4b4aa9b097f33cae154a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.536144 30005 raft_consensus.cc:482] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.536302 30005 raft_consensus.cc:504] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.536991 30005 leader_election.cc:288] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.537467 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7fb65f72628b40a4855474fe07921278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:26.537889 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcc6cef7b7d464a9fbbae61caf0437d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:26.538061 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fb65f72628b40a4855474fe07921278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:26.538552 28150 leader_election.cc:341] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.548986 30142 ts_tablet_manager.cc:1888] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.551753 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c15b4d13647e98cae1375b7f13827 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.554030 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 8fe3affd0d144b1fb0078e7690b9b36d (table default.Table_32b3856710f446139fa555cd42254eef [id=cf15c50bd96d4420b98a06df477003b6]) successfully deleted
I20240503 14:12:26.553755 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet d2eb8328e86a4ee5a53647a810729519 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.554782 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9325d5142af4b018354bf38370b61e0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.555045 28604 ts_tablet_manager.cc:1398] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.130s user 0.003s sys 0.000s
I20240503 14:12:26.555394 28604 tablet_bootstrap.cc:492] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:26.556511 28150 leader_election.cc:341] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.556651 28150 leader_election.cc:302] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.557286 30005 raft_consensus.cc:2726] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.557442 30142 log.cc:1196] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/4fad4508a29d4822b81af6947b57e7ef
I20240503 14:12:26.557911 30142 ts_tablet_manager.cc:1909] T 4fad4508a29d4822b81af6947b57e7ef P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.555410 28661 tablet_replica.cc:330] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.558535 28604 tablet_bootstrap.cc:654] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.558712 28661 raft_consensus.cc:2229] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.559091 28661 raft_consensus.cc:2258] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.559712 28661 ts_tablet_manager.cc:1875] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.562449 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 4fad4508a29d4822b81af6947b57e7ef (table default.Table_68a22cfee7f644fcba9b9c3e34e6205c [id=0a3978674b74406484d7b5e9880f6220]) successfully deleted
I20240503 14:12:26.563024 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 77511682d71a40f5933edcb6a6523649 with delete_type TABLET_DATA_DELETED (replaced by tablet a88cf5de223f4a6abc21691fea1c4da0) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.563283 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.563429 29828 ts_tablet_manager.cc:1909] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:26.563489 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet d2eb8328e86a4ee5a53647a810729519 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:26.563628 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 75029.4us
I20240503 14:12:26.563984 29828 tablet_replica.cc:330] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:26.564152 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c15b4d13647e98cae1375b7f13827 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.564227 29828 raft_consensus.cc:2229] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.564321 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 07354861ee6142d6b1fbe573397d3160 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.564414 29828 raft_consensus.cc:2258] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.564450 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ab077988364bd687e54f90c3c265d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cd037e135904e2cab09cbf39fed0b13) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.565008 29828 ts_tablet_manager.cc:1875] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.565294 28350 raft_consensus.cc:2454] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.567649 30142 tablet_replica.cc:330] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.567903 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50b3e199314bfc8465fdb7b2425d07" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:26.568383 28614 raft_consensus.cc:729] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
W20240503 14:12:26.568558 28150 leader_election.cc:341] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.569319 29994 consensus_queue.cc:1035] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.569869 30142 raft_consensus.cc:2229] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.570176 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ff4d28d3c2ea472191ad21355248a2fc (table default.Table_7b9e454770964e4594e5d043b06c2e78 [id=d4a1074afcca45b287b8d4416878f316]) successfully deleted
I20240503 14:12:26.570989 29994 raft_consensus.cc:482] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.571151 29994 raft_consensus.cc:504] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.571552 30142 raft_consensus.cc:2258] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.571789 29994 leader_election.cc:288] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.572923 30005 raft_consensus.cc:482] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.573063 30005 raft_consensus.cc:504] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.573711 30005 leader_election.cc:288] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:26.574507 28661 ts_tablet_manager.cc:1888] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.574631 28661 log.cc:1196] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/8c6202333af943cbbc0d09553c1c193c
I20240503 14:12:26.574806 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4846d662a64d51b16fd5447c371e9a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:26.575017 28661 ts_tablet_manager.cc:1909] T 8c6202333af943cbbc0d09553c1c193c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:26.575484 28150 leader_election.cc:341] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.575817 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ae1cd28bc07e4ccd9097f6dee471c5c6 (table default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]) successfully deleted
W20240503 14:12:26.576913 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6adeb488fd34301915f2977ce8cf61d consensus state has no leader
I20240503 14:12:26.577167 30142 ts_tablet_manager.cc:1875] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.577356 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5ba631c9bf2f4374991bda5d67c18af0" } new_table_name: "default.Table_d9391d8d03974cf6915092d3fd45e412" modify_external_catalogs: true
I20240503 14:12:26.578634 28661 tablet_replica.cc:330] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.578852 28661 raft_consensus.cc:2229] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.578994 28661 pending_rounds.cc:62] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:26.579030 28661 pending_rounds.cc:68] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597749871136768 op_type: NO_OP noop_request { }
I20240503 14:12:26.579142 28661 raft_consensus.cc:2866] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:26.581817 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_28f6161ffee747d8a270da263c869b2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:26.582113 28661 raft_consensus.cc:2258] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:26.582531 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28f6161ffee747d8a270da263c869b2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.583667 28661 ts_tablet_manager.cc:1875] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.584609 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1ef3157c088942e4ab395d7e11d0a29d" } new_table_name: "default.Table_d25394c1a2084146ad748ab2a35baf46" modify_external_catalogs: true
I20240503 14:12:26.586496 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4d12dbb4326d40ddbeb81be96c45901b" } modify_external_catalogs: true
I20240503 14:12:26.586696 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4d12dbb4326d40ddbeb81be96c45901b" } modify_external_catalogs: true
I20240503 14:12:26.588908 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ebd8ae7785ef42cca8a6b747de4e0285" } modify_external_catalogs: true
I20240503 14:12:26.589119 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ebd8ae7785ef42cca8a6b747de4e0285" } modify_external_catalogs: true
W20240503 14:12:26.590935 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b8c51c0baf44430894429f6eaea797f consensus state has no leader
I20240503 14:12:26.591228 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 6615052ddbde487e840b6fd77a140e58 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:26.594022 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3f8dd5c5498a4ffda480e84d0526e3f4" } modify_external_catalogs: true
I20240503 14:12:26.594280 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3f8dd5c5498a4ffda480e84d0526e3f4" } modify_external_catalogs: true
I20240503 14:12:26.591904 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_26feeadb423f43bcaa8a213736a1e877"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.596138 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26feeadb423f43bcaa8a213736a1e877 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.597630 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7b7ca19dd15f48a68ace7fefc7ae660f
I20240503 14:12:26.598218 28024 catalog_manager.cc:6554] ReplaceTablet: tablet 7b7ca19dd15f48a68ace7fefc7ae660f deleted and replaced by tablet 8ccb01614dee43e48d7e997bcfc7f6a6
I20240503 14:12:26.598981 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7ca19dd15f48a68ace7fefc7ae660f with delete_type TABLET_DATA_DELETED (replaced by tablet 8ccb01614dee43e48d7e997bcfc7f6a6) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.602663 28661 ts_tablet_manager.cc:1888] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.602849 28661 log.cc:1196] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ae1cd28bc07e4ccd9097f6dee471c5c6
I20240503 14:12:26.605357 30142 ts_tablet_manager.cc:1888] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.605842 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.606957 30142 log.cc:1196] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/3cbcae22ea124edaa14c9409f3052714
I20240503 14:12:26.607677 30142 ts_tablet_manager.cc:1909] T 3cbcae22ea124edaa14c9409f3052714 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.612246 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221218847554cbd9d3242a9a6431e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.613685 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcc6cef7b7d464a9fbbae61caf0437d" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.613641 28614 consensus_queue.cc:260] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.614495 29828 ts_tablet_manager.cc:1888] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:26.614526 28150 leader_election.cc:341] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.614645 28150 leader_election.cc:302] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.614688 28661 ts_tablet_manager.cc:1909] T ae1cd28bc07e4ccd9097f6dee471c5c6 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.615056 30132 raft_consensus.cc:2726] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.615511 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4846d662a64d51b16fd5447c371e9a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.615751 30142 tablet_replica.cc:330] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.615754 28661 tablet_replica.cc:330] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.616356 30142 raft_consensus.cc:2229] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.616468 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7ca19dd15f48a68ace7fefc7ae660f with delete_type TABLET_DATA_DELETED (replaced by tablet 8ccb01614dee43e48d7e997bcfc7f6a6) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.617038 28661 raft_consensus.cc:2229] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.617215 28661 raft_consensus.cc:2258] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:26.617308 28150 leader_election.cc:341] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.617406 28150 leader_election.cc:302] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.617758 30132 raft_consensus.cc:2726] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.617081 30142 raft_consensus.cc:2258] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:26.618361 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 429cbd2310c940ad9a8d7a4f4ef17b3b consensus state has no leader
W20240503 14:12:26.619324 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5795755764d9420986aeb10d46f44c45 consensus state has no leader
I20240503 14:12:26.619345 28661 ts_tablet_manager.cc:1875] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.619450 30142 ts_tablet_manager.cc:1875] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.620421 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5b90e9563b4c4cef8ee7964eaa96e285
I20240503 14:12:26.622579 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7d348da459634561985c4ac1710c0bb2
I20240503 14:12:26.623210 28614 ts_tablet_manager.cc:1398] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.136s user 0.002s sys 0.000s
I20240503 14:12:26.623625 28614 tablet_bootstrap.cc:492] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:26.625146 28614 tablet_bootstrap.cc:654] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.626185 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b90e9563b4c4cef8ee7964eaa96e285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.626478 29828 log.cc:1196] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
I20240503 14:12:26.627043 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.627549 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 3cbcae22ea124edaa14c9409f3052714 (table default.Table_6d7202481c4747ddb44a78ba8bb0063f [id=bafd499c743040d08a2fd8207d4f1e2e]) successfully deleted
I20240503 14:12:26.629487 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.629813 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b90e9563b4c4cef8ee7964eaa96e285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:26.638054 28290 consensus_peers.cc:485] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fe3affd0d144b1fb0078e7690b9b36d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:26.645072 30211 consensus_queue.cc:1035] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.646045 28661 ts_tablet_manager.cc:1888] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:26.646086 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c15b4d13647e98cae1375b7f13827 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.646183 28661 log.cc:1196] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0af0342e0e0b47499a9322b5b2cb3d03
I20240503 14:12:26.646342 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4dc732bcb44222b4b15e5d77defe33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:22 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.646551 28661 ts_tablet_manager.cc:1909] T 0af0342e0e0b47499a9322b5b2cb3d03 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.647487 28661 tablet_replica.cc:330] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.647560 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.647702 28661 raft_consensus.cc:2229] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.647804 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 495190fca06a40a386c8215eb2195920 with delete_type TABLET_DATA_DELETED (replaced by tablet 7261088abbc641db8d737a48bbac25bd) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.647884 28661 raft_consensus.cc:2258] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.647936 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ab077988364bd687e54f90c3c265d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cd037e135904e2cab09cbf39fed0b13) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.648051 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.648202 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet dda39f0323ec47799d3d44c96f3ba605 with delete_type TABLET_DATA_DELETED (replaced by tablet e750765798db487fb0b5142f01d21771) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.650597 28604 tablet_bootstrap.cc:492] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.650745 28604 ts_tablet_manager.cc:1367] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.095s user 0.001s sys 0.007s
I20240503 14:12:26.651506 28604 raft_consensus.cc:348] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.651772 28604 raft_consensus.cc:374] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.651826 28604 raft_consensus.cc:729] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.652066 28604 consensus_queue.cc:260] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.652711 28604 ts_tablet_manager.cc:1398] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:26.652983 28604 tablet_bootstrap.cc:492] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.655659 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a544fc27624dcc8a4cb003e59408f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 773bc744579b4ff28c950ea632753221) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.655874 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7ca19dd15f48a68ace7fefc7ae660f with delete_type TABLET_DATA_DELETED (replaced by tablet 8ccb01614dee43e48d7e997bcfc7f6a6) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.656006 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.656154 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet d2eb8328e86a4ee5a53647a810729519 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:25 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.656333 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b90e9563b4c4cef8ee7964eaa96e285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.656528 28661 ts_tablet_manager.cc:1875] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.656530 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a703cfbaad2a4a618abb9e07b4930cee" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:26.656841 28215 raft_consensus.cc:2454] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:26.660513 29828 ts_tablet_manager.cc:1909] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:26.663244 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 8c6202333af943cbbc0d09553c1c193c (table default.Table_fe566740b43c42658f0a0ff2de59038a [id=e9c7ec14573b44b79e70dc70fc9226c2]) successfully deleted
I20240503 14:12:26.663584 28604 tablet_bootstrap.cc:654] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.663887 30142 ts_tablet_manager.cc:1888] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.664191 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a703cfbaad2a4a618abb9e07b4930cee" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.664391 28355 raft_consensus.cc:2454] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:26.665146 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 2f89fca0480d4bf28160384ae0421bce because the tablet was not found. No further retry: Not found: Tablet not found: 2f89fca0480d4bf28160384ae0421bce
I20240503 14:12:26.665408 30142 log.cc:1196] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/b2bba6a3d6e84552b027f23a29cbb38a
I20240503 14:12:26.665527 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ae1cd28bc07e4ccd9097f6dee471c5c6 (table default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]) successfully deleted
I20240503 14:12:26.666477 30142 ts_tablet_manager.cc:1909] T b2bba6a3d6e84552b027f23a29cbb38a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.666769 29828 tablet_replica.cc:330] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:26.667230 30235 tablet.cc:1722] T 64bae128469f4e8bb2d743320731446a P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:26.672379 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5f5dadbbef444ab2b13677a8e8879d4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.673072 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f5dadbbef444ab2b13677a8e8879d4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.673264 28661 ts_tablet_manager.cc:1888] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.673379 28661 log.cc:1196] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/80cd09e769be4bee95bbef9c3c550451
I20240503 14:12:26.673730 28661 ts_tablet_manager.cc:1909] T 80cd09e769be4bee95bbef9c3c550451 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.673892 30142 tablet_replica.cc:330] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.674098 30142 raft_consensus.cc:2229] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.674283 30142 raft_consensus.cc:2258] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.674690 28661 tablet_replica.cc:330] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.674901 28661 raft_consensus.cc:2229] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.675062 28661 raft_consensus.cc:2258] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.675581 28661 ts_tablet_manager.cc:1875] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.675493 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2be14bca3b2b481c93119dbb4ffb6590"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.676076 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be14bca3b2b481c93119dbb4ffb6590 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.676535 30142 ts_tablet_manager.cc:1875] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.677340 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_c9fd8692f46d44adb6c10e08bbccdfa6" } new_table_name: "default.Table_173626c28cf446c4a7d8dcdd69bdc679" modify_external_catalogs: true
I20240503 14:12:26.678915 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet b2bba6a3d6e84552b027f23a29cbb38a (table default.Table_253d4e5a90d94897874db169bc3718b6 [id=ae3e54d70f4541898b7f4d03b4128ac9]) successfully deleted
I20240503 14:12:26.684926 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a (table default.Table_11703c3db99a47cc85741ba32923f695 [id=7448b8a18a6d45c48bc56d1379672494]) successfully deleted
I20240503 14:12:26.685753 30227 tablet.cc:1722] T df63f75c157a4e928d4dbbda06cfa5ce P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240503 14:12:26.691628 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4904b0abf5d644a39079b0667e83aaa0 consensus state has no leader
I20240503 14:12:26.692267 30238 tablet.cc:1722] T 64bae128469f4e8bb2d743320731446a P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:26.694312 30142 ts_tablet_manager.cc:1888] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:26.697712 30142 log.cc:1196] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ae1cd28bc07e4ccd9097f6dee471c5c6
I20240503 14:12:26.698211 30142 ts_tablet_manager.cc:1909] T ae1cd28bc07e4ccd9097f6dee471c5c6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.699142 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 942) took 1017 ms. Trace:
I20240503 14:12:26.699311 30142 rpcz_store.cc:276] 0503 14:12:25.682155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.682196 (+ 41us) service_pool.cc:225] Handling call
0503 14:12:26.673825 (+991629us) ts_tablet_manager.cc:1120] Deleting tablet ae1cd28bc07e4ccd9097f6dee471c5c6
0503 14:12:26.673838 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.686396 (+ 12558us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.694183 (+ 7787us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.699112 (+ 4929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":991487}
I20240503 14:12:26.699505 29828 raft_consensus.cc:2229] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.699980 29828 raft_consensus.cc:2258] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.701581 29828 ts_tablet_manager.cc:1875] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.702528 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ae1cd28bc07e4ccd9097f6dee471c5c6 (table default.Table_222131e1d2a848758bd7e6b5a7498f34 [id=ae8a7a257494489a87e9f5da35395e04]) successfully deleted
I20240503 14:12:26.702999 30142 tablet_replica.cc:330] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.703328 30142 raft_consensus.cc:2229] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.703742 30142 raft_consensus.cc:2258] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.706367 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_75f5b842e7054f3787e97d6641a08485" } modify_external_catalogs: true
I20240503 14:12:26.706624 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_75f5b842e7054f3787e97d6641a08485" } modify_external_catalogs: true
I20240503 14:12:26.708065 28427 leader_election.cc:302] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.708410 30211 raft_consensus.cc:2781] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:26.708489 30211 raft_consensus.cc:482] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:26.708550 30211 raft_consensus.cc:3037] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.704445 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0af0342e0e0b47499a9322b5b2cb3d03 (table default.Table_2c55428ec66a488990244819570ff1bd [id=d160ad3cfaac47c68dcebc3e0b130e5c]) successfully deleted
I20240503 14:12:26.710247 30142 ts_tablet_manager.cc:1875] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.711414 30211 raft_consensus.cc:504] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.712898 30211 leader_election.cc:288] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:26.701021 28600 tablet_bootstrap.cc:492] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:26.716650 28600 ts_tablet_manager.cc:1367] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.252s user 0.011s sys 0.007s
I20240503 14:12:26.717594 28600 raft_consensus.cc:348] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.717828 28600 raft_consensus.cc:374] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.717877 28600 raft_consensus.cc:729] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.718098 28600 consensus_queue.cc:260] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.718830 28600 ts_tablet_manager.cc:1398] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:26.719141 28600 tablet_bootstrap.cc:492] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.720854 28600 tablet_bootstrap.cc:654] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.721323 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_afdf531aa97a45ab9f8c7757d67b3523" } modify_external_catalogs: true
I20240503 14:12:26.721587 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_afdf531aa97a45ab9f8c7757d67b3523" } modify_external_catalogs: true
I20240503 14:12:26.723712 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_41235c2e757a4d39925d2d8e6dc69514"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.724588 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41235c2e757a4d39925d2d8e6dc69514 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.731035 30142 ts_tablet_manager.cc:1888] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.731187 30142 log.cc:1196] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/920be6d87c4440d0bf7453c86cfe0b5c
I20240503 14:12:26.731402 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 80cd09e769be4bee95bbef9c3c550451 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
I20240503 14:12:26.733975 30142 ts_tablet_manager.cc:1909] T 920be6d87c4440d0bf7453c86cfe0b5c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.735551 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a703cfbaad2a4a618abb9e07b4930cee" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:26.736256 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a703cfbaad2a4a618abb9e07b4930cee" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:26.736464 28352 raft_consensus.cc:3037] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.736495 28215 raft_consensus.cc:3037] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.739133 28215 raft_consensus.cc:2454] T a703cfbaad2a4a618abb9e07b4930cee P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.739151 28352 raft_consensus.cc:2454] T a703cfbaad2a4a618abb9e07b4930cee P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.741786 30211 raft_consensus.cc:482] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.741979 30211 raft_consensus.cc:504] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.742642 30211 leader_election.cc:288] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:26.743671 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 943) took 1061 ms. Trace:
I20240503 14:12:26.744472 30142 rpcz_store.cc:276] 0503 14:12:25.682647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.682700 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:26.702932 (+1020232us) ts_tablet_manager.cc:1120] Deleting tablet 920be6d87c4440d0bf7453c86cfe0b5c
0503 14:12:26.702945 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.723983 (+ 21038us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.730908 (+ 6925us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.743643 (+ 12735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020091}
I20240503 14:12:26.745913 28614 tablet_bootstrap.cc:492] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:26.746083 28614 ts_tablet_manager.cc:1367] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.000s
I20240503 14:12:26.746155 28604 tablet_bootstrap.cc:492] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.746331 28604 ts_tablet_manager.cc:1367] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.093s user 0.000s sys 0.002s
I20240503 14:12:26.746829 28614 raft_consensus.cc:348] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.747049 28614 raft_consensus.cc:374] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.747104 28614 raft_consensus.cc:729] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.747385 28614 consensus_queue.cc:260] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.747547 28604 raft_consensus.cc:348] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.747682 30107 raft_consensus.cc:482] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.747756 28604 raft_consensus.cc:374] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.747822 28604 raft_consensus.cc:729] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.747790 30107 raft_consensus.cc:504] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.748018 28604 consensus_queue.cc:260] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.748452 30107 leader_election.cc:288] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:26.748941 28604 ts_tablet_manager.cc:1398] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:26.749146 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 920be6d87c4440d0bf7453c86cfe0b5c (table default.Table_d8cb0703edfb4e96ba081bd6f299d867 [id=6119a6b076964647bd7247e0a896a030]) successfully deleted
I20240503 14:12:26.749264 28604 tablet_bootstrap.cc:492] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.749531 30142 tablet_replica.cc:330] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.749753 30142 raft_consensus.cc:2229] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.750046 30142 raft_consensus.cc:2258] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.750777 28604 tablet_bootstrap.cc:654] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.750782 30142 ts_tablet_manager.cc:1875] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.752486 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69425d29d55b4fe4bb7f08ae390aaa86" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:26.752743 28493 raft_consensus.cc:2454] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:26.752669 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69425d29d55b4fe4bb7f08ae390aaa86" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:26.752882 28213 raft_consensus.cc:2454] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:26.754256 30211 consensus_queue.cc:1035] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.754910 28290 leader_election.cc:302] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:26.755087 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 728ba3ac1cb645168481b84b5b548140
I20240503 14:12:26.756655 30107 raft_consensus.cc:2781] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:26.756750 30107 raft_consensus.cc:482] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:26.756807 30107 raft_consensus.cc:3037] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.756764 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 728ba3ac1cb645168481b84b5b548140 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.758256 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 728ba3ac1cb645168481b84b5b548140 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.759400 30107 raft_consensus.cc:504] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.760151 30142 ts_tablet_manager.cc:1888] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.760895 30107 leader_election.cc:288] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:26.761543 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 64bae128469f4e8bb2d743320731446a (table default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f]). Expected version 1 got 0
I20240503 14:12:26.761639 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69425d29d55b4fe4bb7f08ae390aaa86" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:26.761835 28493 raft_consensus.cc:3037] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.762107 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:26.763566 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_bfecf31f8b1443d58376a8125ebcbff2 [id=d29fe68058a14e178e42a10043a91738] alter complete (version 4)
I20240503 14:12:26.765081 28493 raft_consensus.cc:2454] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:26.765894 28290 leader_election.cc:302] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.766286 30107 raft_consensus.cc:2781] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.766541 30107 raft_consensus.cc:686] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:26.766937 30107 consensus_queue.cc:237] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.761639 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69425d29d55b4fe4bb7f08ae390aaa86" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:26.768682 28213 raft_consensus.cc:3037] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.769277 28326 tablet_service.cc:1467] Processing CreateTablet for tablet df2e08278389408f903f7b4709e41445 (DEFAULT_TABLE table=default.Table_4f391faa840c44368eba424683091351 [id=e4a20e7619bf402dbbd5a7b88c0742b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.769687 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2e08278389408f903f7b4709e41445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.771487 28213 raft_consensus.cc:2454] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:26.766455 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbcae22ea124edaa14c9409f3052714 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.766626 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084bd4fbf604042bb33ff03c3f8cc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.766752 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.771862 28614 ts_tablet_manager.cc:1398] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 14:12:26.766853 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.766940 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.767025 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.772042 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 778ddcb861f341d4bebe7ddd08ef36b1 (DEFAULT_TABLE table=default.Table_6349dd55828a4c548a0472ada6ffdc73 [id=ca08f7ca3ca9423f8693683a5826abec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.772611 30142 log.cc:1196] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/d43823da09bb49be8f5d1a1149a4b11d
I20240503 14:12:26.772459 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778ddcb861f341d4bebe7ddd08ef36b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.773344 30142 ts_tablet_manager.cc:1909] T d43823da09bb49be8f5d1a1149a4b11d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.773834 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 778ddcb861f341d4bebe7ddd08ef36b1 (DEFAULT_TABLE table=default.Table_6349dd55828a4c548a0472ada6ffdc73 [id=ca08f7ca3ca9423f8693683a5826abec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.773495 30132 raft_consensus.cc:482] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.774165 30132 raft_consensus.cc:504] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.774859 30132 leader_election.cc:288] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:26.775804 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778ddcb861f341d4bebe7ddd08ef36b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.776129 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet bab214b7ec58425d8cd1180ee16b273d
I20240503 14:12:26.776175 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 944) took 1093 ms. Trace:
I20240503 14:12:26.776255 30142 rpcz_store.cc:276] 0503 14:12:25.683052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.683091 (+ 39us) service_pool.cc:225] Handling call
0503 14:12:26.749469 (+1066378us) ts_tablet_manager.cc:1120] Deleting tablet d43823da09bb49be8f5d1a1149a4b11d
0503 14:12:26.749480 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.757098 (+ 7618us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.760010 (+ 2912us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.776149 (+ 16139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1066242}
I20240503 14:12:26.776779 30142 tablet_replica.cc:330] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:26.776964 30142 raft_consensus.cc:2229] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.777097 30142 raft_consensus.cc:2258] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.777602 30142 ts_tablet_manager.cc:1875] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.772203 28614 tablet_bootstrap.cc:492] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:26.781229 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet d43823da09bb49be8f5d1a1149a4b11d (table default.Table_e65026502109470c9382acbdf03874d6 [id=99f3436cf65347648070d088f8430e1a]) successfully deleted
I20240503 14:12:26.781841 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet bab214b7ec58425d8cd1180ee16b273d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.782326 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_85c71baa763440bf939ea1193fa0af0c" } new_table_name: "default.Table_a7c0a974ecd14e0286633f49da7d3741" modify_external_catalogs: true
I20240503 14:12:26.782656 30227 tablet.cc:1722] T 64bae128469f4e8bb2d743320731446a P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:26.787026 30142 ts_tablet_manager.cc:1888] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:26.787170 30142 log.cc:1196] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/07354861ee6142d6b1fbe573397d3160
I20240503 14:12:26.787497 30142 ts_tablet_manager.cc:1909] T 07354861ee6142d6b1fbe573397d3160 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:26.787503 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet c725628f72884086b523b56ecf876069
I20240503 14:12:26.788348 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 945) took 1051 ms. Trace:
I20240503 14:12:26.788364 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet bab214b7ec58425d8cd1180ee16b273d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.790462 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 86d810ab0cf34941a343eef1593632bf (table default.Table_9f2d3cc9c6724dab9ba35ff24e85c236 [id=c358e28088d44cf598c96a1bd2a4321e]). Expected version 1 got 0
I20240503 14:12:26.793566 28326 tablet_service.cc:1467] Processing CreateTablet for tablet bfc28afd18b849fbb23e8b71e932516d (DEFAULT_TABLE table=default.Table_851943f2c8df437dbfc6a677f5ef4900 [id=802d294f97cb4e82b65af3e966e09472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.793830 28321 tablet_service.cc:1467] Processing CreateTablet for tablet ec64e66fcdf6452ea65688a36fd07fde (DEFAULT_TABLE table=default.Table_f9bcc668aff94b09927c733f071bc6de [id=0fc0e47dacc84ac7b47d5f8cac67dc8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.794008 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc28afd18b849fbb23e8b71e932516d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.794193 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec64e66fcdf6452ea65688a36fd07fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.796495 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet c725628f72884086b523b56ecf876069 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.790944 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4cd19da5bf64494e8ff53ded46bc5c4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.798759 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cd19da5bf64494e8ff53ded46bc5c4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.800395 28335 tablet_service.cc:1467] Processing CreateTablet for tablet be58d7b6fdf24bf1ad421182b660c77c (DEFAULT_TABLE table=default.Table_ac11e443561a400ea0c013dd4e9178cf [id=af3dd98a87204d7a89b5a09ecfcf3ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.801853 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be58d7b6fdf24bf1ad421182b660c77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.802628 29828 ts_tablet_manager.cc:1888] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.802724 29828 log.cc:1196] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ff1958dc488e4854a8c8e74da9fab243
I20240503 14:12:26.804164 28614 tablet_bootstrap.cc:654] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.796748 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6a7863d34f3a47f38c253755f3089f6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.804965 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a7863d34f3a47f38c253755f3089f6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.797176 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3d0d6e04014b414aa95434036f1b4516"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.805801 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d0d6e04014b414aa95434036f1b4516 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.797508 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d9391d8d03974cf6915092d3fd45e412" } new_table_name: "default.Table_e6202e9cd7bb447da2a1412d4a4a11bd" modify_external_catalogs: true
I20240503 14:12:26.808480 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c9fcd1ad02784a29858da8e74a9c9866"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.809093 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9fcd1ad02784a29858da8e74a9c9866 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.793447 28196 tablet_service.cc:1467] Processing CreateTablet for tablet bfc28afd18b849fbb23e8b71e932516d (DEFAULT_TABLE table=default.Table_851943f2c8df437dbfc6a677f5ef4900 [id=802d294f97cb4e82b65af3e966e09472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.810876 29828 ts_tablet_manager.cc:1909] T ff1958dc488e4854a8c8e74da9fab243 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:26.811118 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc28afd18b849fbb23e8b71e932516d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.811977 29828 tablet_replica.cc:330] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:26.812211 29828 raft_consensus.cc:2229] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:26.812528 29828 raft_consensus.cc:2258] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.798156 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2bc25daca38a439a9c3275f9cb988224"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.813459 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bc25daca38a439a9c3275f9cb988224 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.813756 29828 ts_tablet_manager.cc:1875] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.815819 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 60ef46a8638248a9b9c46e48fbb8cfcc (DEFAULT_TABLE table=default.Table_16dc72b61e7248c191c1daf2d14f63a9 [id=ce80aae3983c43dfbc6786dbcf5849e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.816052 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 75c16c31fea74d99980b2e101efc3f36 (DEFAULT_TABLE table=default.Table_9dfb63e159c34cb9a11a9a5976aa4979 [id=b5cea63330704d67b9c3a5abfa0a1e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.816308 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ef46a8638248a9b9c46e48fbb8cfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.816481 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c16c31fea74d99980b2e101efc3f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.817055 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d1314841574fc8bcf3ccb16755e421" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:26.817639 28150 leader_election.cc:341] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:26.817703 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 63521.8us
I20240503 14:12:26.817946 28427 leader_election.cc:302] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.818651 29994 raft_consensus.cc:2781] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.819365 29994 raft_consensus.cc:686] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:26.819536 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ff1958dc488e4854a8c8e74da9fab243 (table default.Table_8ea2f91bb9e242aeb07b426cd711d537 [id=aed23d17e9704c36b4b4ecb604987e37]) successfully deleted
I20240503 14:12:26.819644 29994 consensus_queue.cc:237] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.815819 28186 tablet_service.cc:1467] Processing CreateTablet for tablet d479f4f392244d5c877db4820367b224 (DEFAULT_TABLE table=default.Table_6331f7c5c3804f44833a92e6ded96825 [id=64e016a2535a4e88a30f199c8ab99bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.821235 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d479f4f392244d5c877db4820367b224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.822510 28192 tablet_service.cc:1467] Processing CreateTablet for tablet c9bf4788fa784f49bb2fa4de79d8064a (DEFAULT_TABLE table=default.Table_a77dd1752a4843739a371c414093b2c8 [id=572237d3a5784381bfc9afe4f2b2460e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.822670 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 1906ab252ea349cfa0afbd9f4303efbc (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.822885 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bf4788fa784f49bb2fa4de79d8064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.823051 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1906ab252ea349cfa0afbd9f4303efbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.823773 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet bab214b7ec58425d8cd1180ee16b273d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.823992 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 728ba3ac1cb645168481b84b5b548140 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.824357 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 778ddcb861f341d4bebe7ddd08ef36b1 (DEFAULT_TABLE table=default.Table_6349dd55828a4c548a0472ada6ffdc73 [id=ca08f7ca3ca9423f8693683a5826abec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.824769 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778ddcb861f341d4bebe7ddd08ef36b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.826643 28459 tablet_service.cc:1467] Processing CreateTablet for tablet ec64e66fcdf6452ea65688a36fd07fde (DEFAULT_TABLE table=default.Table_f9bcc668aff94b09927c733f071bc6de [id=0fc0e47dacc84ac7b47d5f8cac67dc8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.827013 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec64e66fcdf6452ea65688a36fd07fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.827698 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet c725628f72884086b523b56ecf876069 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.828119 28466 tablet_service.cc:1467] Processing CreateTablet for tablet df2e08278389408f903f7b4709e41445 (DEFAULT_TABLE table=default.Table_4f391faa840c44368eba424683091351 [id=e4a20e7619bf402dbbd5a7b88c0742b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.828606 28457 tablet_service.cc:1467] Processing CreateTablet for tablet bfc28afd18b849fbb23e8b71e932516d (DEFAULT_TABLE table=default.Table_851943f2c8df437dbfc6a677f5ef4900 [id=802d294f97cb4e82b65af3e966e09472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.829030 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 60ef46a8638248a9b9c46e48fbb8cfcc (DEFAULT_TABLE table=default.Table_16dc72b61e7248c191c1daf2d14f63a9 [id=ce80aae3983c43dfbc6786dbcf5849e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.829445 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 75c16c31fea74d99980b2e101efc3f36 (DEFAULT_TABLE table=default.Table_9dfb63e159c34cb9a11a9a5976aa4979 [id=b5cea63330704d67b9c3a5abfa0a1e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.829512 28472 tablet_service.cc:1467] Processing CreateTablet for tablet d479f4f392244d5c877db4820367b224 (DEFAULT_TABLE table=default.Table_6331f7c5c3804f44833a92e6ded96825 [id=64e016a2535a4e88a30f199c8ab99bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.829998 28456 tablet_service.cc:1467] Processing CreateTablet for tablet c9bf4788fa784f49bb2fa4de79d8064a (DEFAULT_TABLE table=default.Table_a77dd1752a4843739a371c414093b2c8 [id=572237d3a5784381bfc9afe4f2b2460e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.830694 28461 tablet_service.cc:1467] Processing CreateTablet for tablet be58d7b6fdf24bf1ad421182b660c77c (DEFAULT_TABLE table=default.Table_ac11e443561a400ea0c013dd4e9178cf [id=af3dd98a87204d7a89b5a09ecfcf3ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.830899 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e9d878dfb144e5db914f8619c13ce8c) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.831115 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 1906ab252ea349cfa0afbd9f4303efbc (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.831430 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.831454 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9325d5142af4b018354bf38370b61e0" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:26.831599 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet d93fea5caec94695a5df26d3630c50c7 with delete_type TABLET_DATA_DELETED (replaced by tablet b476e87faea1415d95ae014a93fa22af) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.831692 28213 raft_consensus.cc:2454] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:26.832183 28468 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.832857 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:26.793601 28187 tablet_service.cc:1467] Processing CreateTablet for tablet df2e08278389408f903f7b4709e41445 (DEFAULT_TABLE table=default.Table_4f391faa840c44368eba424683091351 [id=e4a20e7619bf402dbbd5a7b88c0742b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.833256 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2e08278389408f903f7b4709e41445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.834303 28600 tablet_bootstrap.cc:492] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:26.834432 28600 ts_tablet_manager.cc:1367] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.115s user 0.003s sys 0.000s
I20240503 14:12:26.835598 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 75c16c31fea74d99980b2e101efc3f36 (DEFAULT_TABLE table=default.Table_9dfb63e159c34cb9a11a9a5976aa4979 [id=b5cea63330704d67b9c3a5abfa0a1e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.835673 28318 tablet_service.cc:1467] Processing CreateTablet for tablet c9bf4788fa784f49bb2fa4de79d8064a (DEFAULT_TABLE table=default.Table_a77dd1752a4843739a371c414093b2c8 [id=572237d3a5784381bfc9afe4f2b2460e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.836050 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c16c31fea74d99980b2e101efc3f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.836263 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bf4788fa784f49bb2fa4de79d8064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.836411 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 1906ab252ea349cfa0afbd9f4303efbc (DEFAULT_TABLE table=default.Table_d933cabbaf9146438f67771682480e59 [id=b086c0d2ac7140acab5b95e9a15842e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.836751 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1906ab252ea349cfa0afbd9f4303efbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.795507 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet c725628f72884086b523b56ecf876069 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.800868 28180 tablet_service.cc:1467] Processing CreateTablet for tablet be58d7b6fdf24bf1ad421182b660c77c (DEFAULT_TABLE table=default.Table_ac11e443561a400ea0c013dd4e9178cf [id=af3dd98a87204d7a89b5a09ecfcf3ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.840965 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9325d5142af4b018354bf38370b61e0" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.799997 28179 tablet_service.cc:1467] Processing CreateTablet for tablet ec64e66fcdf6452ea65688a36fd07fde (DEFAULT_TABLE table=default.Table_f9bcc668aff94b09927c733f071bc6de [id=0fc0e47dacc84ac7b47d5f8cac67dc8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.841245 28025 catalog_manager.cc:5484] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:26.841828 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc28afd18b849fbb23e8b71e932516d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.841744 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 77c77a11ef384709aef5b62273380573 (table default.Table_ab495b10ad82425ba5c7123a6e1cf064 [id=2edd1164c07a4a67b51262d402d53596]). Expected version 2 got 0
I20240503 14:12:26.842432 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bf4788fa784f49bb2fa4de79d8064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.842401 28025 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:26.843349 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c16c31fea74d99980b2e101efc3f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.843883 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ef46a8638248a9b9c46e48fbb8cfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.844434 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2e08278389408f903f7b4709e41445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.843971 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 60ef46a8638248a9b9c46e48fbb8cfcc (DEFAULT_TABLE table=default.Table_16dc72b61e7248c191c1daf2d14f63a9 [id=ce80aae3983c43dfbc6786dbcf5849e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:26.845861 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d479f4f392244d5c877db4820367b224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.845830 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ef46a8638248a9b9c46e48fbb8cfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.846426 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1906ab252ea349cfa0afbd9f4303efbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.848516 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be58d7b6fdf24bf1ad421182b660c77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.850181 28661 ts_tablet_manager.cc:1888] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.850304 28661 log.cc:1196] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6d6b50aed01a4bf8af8889036dd83606
I20240503 14:12:26.850634 28661 ts_tablet_manager.cc:1909] T 6d6b50aed01a4bf8af8889036dd83606 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.854270 28335 tablet_service.cc:1467] Processing CreateTablet for tablet d479f4f392244d5c877db4820367b224 (DEFAULT_TABLE table=default.Table_6331f7c5c3804f44833a92e6ded96825 [id=64e016a2535a4e88a30f199c8ab99bd0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:26.864478 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b008e3b655644d4a4f998df1fc7c1c9 consensus state has no leader
I20240503 14:12:26.865001 28604 tablet_bootstrap.cc:492] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.865036 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d25394c1a2084146ad748ab2a35baf46" } new_table_name: "default.Table_5ac715bff6194f9abbdc93e6e59d77e7" modify_external_catalogs: true
I20240503 14:12:26.865906 28604 ts_tablet_manager.cc:1367] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.117s user 0.003s sys 0.001s
I20240503 14:12:26.866984 28493 raft_consensus.cc:3037] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.847028 30005 raft_consensus.cc:482] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.840607 28600 raft_consensus.cc:348] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.867496 30005 raft_consensus.cc:504] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.867591 28600 raft_consensus.cc:374] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.867642 28600 raft_consensus.cc:729] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.867851 28600 consensus_queue.cc:260] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.841351 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be58d7b6fdf24bf1ad421182b660c77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.868031 28604 raft_consensus.cc:348] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.868250 28604 raft_consensus.cc:374] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.868299 28604 raft_consensus.cc:729] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.868467 28604 consensus_queue.cc:260] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:26.868562 28600 ts_tablet_manager.cc:1398] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.034s user 0.000s sys 0.002s
I20240503 14:12:26.868856 28600 tablet_bootstrap.cc:492] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:26.869130 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec64e66fcdf6452ea65688a36fd07fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.869920 30005 leader_election.cc:288] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:26.870504 28600 tablet_bootstrap.cc:654] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.870575 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_39c390a1cebb4679b78b0e36750364d6" } modify_external_catalogs: true
I20240503 14:12:26.870781 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_39c390a1cebb4679b78b0e36750364d6" } modify_external_catalogs: true
I20240503 14:12:26.871232 30005 consensus_queue.cc:1035] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:26.872792 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 429cbd2310c940ad9a8d7a4f4ef17b3b consensus state has no leader
I20240503 14:12:26.873277 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1c8ad5b7f0304d83a8cdf65bff9e76e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.873857 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c8ad5b7f0304d83a8cdf65bff9e76e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.874769 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 969) took 1198 ms. Trace:
I20240503 14:12:26.875191 28661 rpcz_store.cc:276] 0503 14:12:25.676413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.676776 (+ 363us) service_pool.cc:225] Handling call
0503 14:12:26.674640 (+997864us) ts_tablet_manager.cc:1120] Deleting tablet 6d6b50aed01a4bf8af8889036dd83606
0503 14:12:26.674649 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.703359 (+ 28710us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.850069 (+146710us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.874740 (+ 24671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":997543}
I20240503 14:12:26.877308 28661 tablet_replica.cc:330] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:26.877315 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e5c58148b8e649f482d8025f0d340f3b" } modify_external_catalogs: true
I20240503 14:12:26.877535 28661 raft_consensus.cc:2229] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:26.877532 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e5c58148b8e649f482d8025f0d340f3b" } modify_external_catalogs: true
I20240503 14:12:26.877700 28661 raft_consensus.cc:2258] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:26.878511 28604 ts_tablet_manager.cc:1398] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:26.878829 28604 tablet_bootstrap.cc:492] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.880036 28661 ts_tablet_manager.cc:1875] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:26.879854 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_33a3711d0f824b1499cb143a33f44014"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.880563 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33a3711d0f824b1499cb143a33f44014 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.881464 30107 raft_consensus.cc:482] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:26.881616 30107 raft_consensus.cc:504] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.882340 30107 leader_election.cc:288] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:26.883052 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:26.883765 28290 leader_election.cc:341] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.884294 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:26.884524 28215 raft_consensus.cc:2454] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:26.885764 28290 leader_election.cc:302] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters: d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.886130 30107 raft_consensus.cc:2781] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:26.886209 30107 raft_consensus.cc:482] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:26.886286 30107 raft_consensus.cc:3037] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.888077 28604 tablet_bootstrap.cc:654] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:26.888262 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:26.889225 30107 raft_consensus.cc:504] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.889914 30107 leader_election.cc:288] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:26.892740 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:26.892982 28215 raft_consensus.cc:3037] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.893636 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1ddf62892e2149ccb722a65a606c43e7" } new_table_name: "default.Table_ccc2979c8006441e888f15858f2c06d4" modify_external_catalogs: true
I20240503 14:12:26.895661 28215 raft_consensus.cc:2454] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:26.896448 28290 leader_election.cc:302] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33; no voters:
I20240503 14:12:26.897169 30107 raft_consensus.cc:2781] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.898845 30107 raft_consensus.cc:686] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:26.899542 30107 consensus_queue.cc:237] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:26.903962 28150 leader_election.cc:341] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.904465 28352 raft_consensus.cc:3037] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.905901 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b517009a45264b2eb9d5c01512efabc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.906616 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b517009a45264b2eb9d5c01512efabc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.906675 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_c1e38c13e56749f990b2f0d9c5863be5 [id=cb2015955266483fa60f3fc692731e5f] alter complete (version 1)
I20240503 14:12:26.908401 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d1314841574fc8bcf3ccb16755e421" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.909649 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4846d662a64d51b16fd5447c371e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.910344 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d1314841574fc8bcf3ccb16755e421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.910254 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7551243835a942e4ac44d0f728b28fe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:26.911010 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7551243835a942e4ac44d0f728b28fe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:26.911031 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.911559 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df with delete_type TABLET_DATA_DELETED (replaced by tablet ae909ab6263a401490952e3f68533741) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.913208 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.916159 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.920032 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:26.924971 28604 tablet_bootstrap.cc:492] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:26.925082 28427 leader_election.cc:302] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.925153 28604 ts_tablet_manager.cc:1367] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
I20240503 14:12:26.925359 30211 raft_consensus.cc:2781] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:26.925463 30211 raft_consensus.cc:482] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:26.925529 30211 raft_consensus.cc:3037] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.925501 29996 consensus_queue.cc:1035] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.925835 28604 raft_consensus.cc:348] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.926026 28604 raft_consensus.cc:374] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:26.926079 28604 raft_consensus.cc:729] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.931550 28604 consensus_queue.cc:260] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.932345 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6d6b50aed01a4bf8af8889036dd83606 (table default.Table_ede6e833710649e08e31516f0890c2d6 [id=3d49172b73ba44538840b651a85b24e3]) successfully deleted
I20240503 14:12:26.932591 30005 consensus_queue.cc:1035] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:26.935700 30211 raft_consensus.cc:504] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.937100 30211 leader_election.cc:288] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:26.939623 28427 leader_election.cc:341] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.942749 28604 ts_tablet_manager.cc:1398] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20240503 14:12:26.942988 30107 consensus_queue.cc:1035] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:26.950258 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57a959b76bc847ad82fbc847d9947c16 consensus state has no leader
I20240503 14:12:26.950873 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9d113544122942a38f6c98e899f8c488"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:26.951028 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:26.951524 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d113544122942a38f6c98e899f8c488 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:26.951746 28290 leader_election.cc:341] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.953429 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d479f4f392244d5c877db4820367b224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.953444 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9db657eef2db4e3791318e01e1a965fb
I20240503 14:12:26.953682 28029 catalog_manager.cc:5484] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a6b9b7913444d29c96581af6febf33 (127.27.50.194). New cstate: current_term: 1 leader_uuid: "d3a6b9b7913444d29c96581af6febf33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:26.954197 28029 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
W20240503 14:12:26.955575 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5795755764d9420986aeb10d46f44c45 consensus state has no leader
I20240503 14:12:26.957355 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5ac145bcc44b4437b840df389cd254f9
I20240503 14:12:26.957798 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc3d602d565c4c32b9127717dc19a658" } modify_external_catalogs: true
I20240503 14:12:26.958053 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cc3d602d565c4c32b9127717dc19a658" } modify_external_catalogs: true
I20240503 14:12:26.961038 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1958dc488e4854a8c8e74da9fab243 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:14 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.967128 28604 tablet_bootstrap.cc:492] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:26.969445 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9325d5142af4b018354bf38370b61e0" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:26.969714 28214 raft_consensus.cc:3037] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:26.969982 30005 consensus_queue.cc:1035] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:26.970863 28150 leader_election.cc:341] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.971087 28150 leader_election.cc:302] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.971272 28661 ts_tablet_manager.cc:1888] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:26.971400 30005 raft_consensus.cc:2726] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.971737 28661 log.cc:1196] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/d43823da09bb49be8f5d1a1149a4b11d
W20240503 14:12:26.971913 28150 leader_election.cc:341] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.972213 28150 leader_election.cc:302] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:26.972512 30005 raft_consensus.cc:2726] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:26.972538 28661 ts_tablet_manager.cc:1909] T d43823da09bb49be8f5d1a1149a4b11d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:26.972595 28214 raft_consensus.cc:2454] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:26.973918 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 5500da0e5658492b840dd29273ab265f from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:26.974207 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9325d5142af4b018354bf38370b61e0" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:26.975026 28427 consensus_peers.cc:485] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07354861ee6142d6b1fbe573397d3160. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:26.975224 28427 leader_election.cc:302] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:26.975486 29996 raft_consensus.cc:2781] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:26.975610 29996 raft_consensus.cc:686] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:26.975858 29996 consensus_queue.cc:237] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:26.979089 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_eb65ad8d07974a069b16d09a5dd36d1d" } modify_external_catalogs: true
I20240503 14:12:26.979375 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_eb65ad8d07974a069b16d09a5dd36d1d" } modify_external_catalogs: true
I20240503 14:12:26.979410 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4d29fd37826542f6aee3139a6338e4ae" } new_table_name: "default.Table_e13134ed06ba4311ac8738052af0fe83" modify_external_catalogs: true
I20240503 14:12:26.982424 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db657eef2db4e3791318e01e1a965fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.985834 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac145bcc44b4437b840df389cd254f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.986265 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac145bcc44b4437b840df389cd254f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:26.987529 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db657eef2db4e3791318e01e1a965fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:26.989704 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_55224d5639254fa1a4890def9934e40a" } modify_external_catalogs: true
I20240503 14:12:26.989991 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_55224d5639254fa1a4890def9934e40a" } modify_external_catalogs: true
I20240503 14:12:26.991557 30211 consensus_queue.cc:1035] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:26.992166 28427 leader_election.cc:341] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:26.996385 30211 consensus_queue.cc:1035] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.000027 29828 ts_tablet_manager.cc:1888] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.004019 30142 rpcz_store.cc:276] 0503 14:12:25.736947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.737015 (+ 68us) service_pool.cc:225] Handling call
0503 14:12:26.776719 (+1039704us) ts_tablet_manager.cc:1120] Deleting tablet 07354861ee6142d6b1fbe573397d3160
0503 14:12:26.776737 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.783943 (+ 7206us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.786904 (+ 2961us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.788313 (+ 1409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039455}
W20240503 14:12:27.004596 28427 consensus_peers.cc:485] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0fd8fb8ea1a4fc3bfa4f55ea59ae12a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.013782 28604 tablet_bootstrap.cc:654] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.015048 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 971) took 1338 ms. Trace:
I20240503 14:12:27.015164 28661 rpcz_store.cc:276] 0503 14:12:25.676584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.677042 (+ 458us) service_pool.cc:225] Handling call
0503 14:12:26.877243 (+1200201us) ts_tablet_manager.cc:1120] Deleting tablet d43823da09bb49be8f5d1a1149a4b11d
0503 14:12:26.877257 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.922691 (+ 45434us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.967944 (+ 45253us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.015015 (+ 47071us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198359}
I20240503 14:12:27.015515 28661 tablet_replica.cc:330] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.015836 28661 raft_consensus.cc:2229] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.016021 28661 raft_consensus.cc:2258] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.020838 28661 ts_tablet_manager.cc:1875] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.021430 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac145bcc44b4437b840df389cd254f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.021596 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db657eef2db4e3791318e01e1a965fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.024940 28604 tablet_bootstrap.cc:492] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.025722 30005 consensus_queue.cc:1035] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.026475 28604 ts_tablet_manager.cc:1367] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.005s
I20240503 14:12:27.027174 28600 tablet_bootstrap.cc:492] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.027110 30238 tablet.cc:1722] T b75036cdf9e04ea4aa767fd12016f177 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:27.026433 30235 tablet.cc:1722] T b75036cdf9e04ea4aa767fd12016f177 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:27.027665 28600 ts_tablet_manager.cc:1367] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.159s user 0.002s sys 0.002s
I20240503 14:12:27.028410 28600 raft_consensus.cc:348] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.028616 28600 raft_consensus.cc:374] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.028661 28600 raft_consensus.cc:729] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.028865 28600 consensus_queue.cc:260] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:27.036758 28150 consensus_peers.cc:485] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d93fea5caec94695a5df26d3630c50c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.032305 28604 raft_consensus.cc:348] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.017845 30142 tablet_replica.cc:330] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.038682 30142 raft_consensus.cc:2229] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.038937 28604 raft_consensus.cc:374] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.040683 28604 raft_consensus.cc:729] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.062880 30142 raft_consensus.cc:2258] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.063030 28600 ts_tablet_manager.cc:1398] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240503 14:12:27.063360 28600 tablet_bootstrap.cc:492] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.063953 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 07354861ee6142d6b1fbe573397d3160 (table default.Table_7fbc01371cd145eeb7b51110d14f6456 [id=2cc7a41775044f4a993bd36fc229d152]) successfully deleted
I20240503 14:12:27.064836 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b41e3b21347b393497de4d4e87469 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.065605 30142 ts_tablet_manager.cc:1875] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.063218 28604 consensus_queue.cc:260] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.071015 28604 ts_tablet_manager.cc:1398] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240503 14:12:27.075523 30134 raft_consensus.cc:482] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.076244 30134 raft_consensus.cc:504] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.076426 28604 tablet_bootstrap.cc:492] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.076964 30134 leader_election.cc:288] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.078449 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.078770 28212 raft_consensus.cc:2454] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.079403 28427 leader_election.cc:302] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.079821 30134 raft_consensus.cc:2781] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.079903 30134 raft_consensus.cc:482] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.079962 30134 raft_consensus.cc:3037] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.080972 28600 tablet_bootstrap.cc:654] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.084322 28604 tablet_bootstrap.cc:654] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.087543 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ab495b10ad82425ba5c7123a6e1cf064" } new_table_name: "default.Table_ece674e6e74547b6a4d04f1f2b167a08" modify_external_catalogs: true
I20240503 14:12:27.089033 30134 raft_consensus.cc:504] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.090173 30134 leader_election.cc:288] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.085878 28661 ts_tablet_manager.cc:1888] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.092460 28661 log.cc:1196] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/4084bd4fbf604042bb33ff03c3f8cc19
I20240503 14:12:27.093034 28661 ts_tablet_manager.cc:1909] T 4084bd4fbf604042bb33ff03c3f8cc19 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.094771 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 974) took 1417 ms. Trace:
I20240503 14:12:27.097744 28661 rpcz_store.cc:276] 0503 14:12:25.676918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.677113 (+ 195us) service_pool.cc:225] Handling call
0503 14:12:27.015458 (+1338345us) ts_tablet_manager.cc:1120] Deleting tablet 4084bd4fbf604042bb33ff03c3f8cc19
0503 14:12:27.015469 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.068148 (+ 52679us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.085722 (+ 17574us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.094739 (+ 9017us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1338222}
I20240503 14:12:27.098492 28661 tablet_replica.cc:330] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.098987 28661 raft_consensus.cc:2229] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.099253 28661 raft_consensus.cc:2258] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.099478 30142 ts_tablet_manager.cc:1888] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.100004 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet d43823da09bb49be8f5d1a1149a4b11d (table default.Table_e65026502109470c9382acbdf03874d6 [id=99f3436cf65347648070d088f8430e1a]) successfully deleted
I20240503 14:12:27.100220 28661 ts_tablet_manager.cc:1875] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.100435 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 4084bd4fbf604042bb33ff03c3f8cc19 (table default.Table_3a0134977dde48a0849c29c36487c742 [id=119495cd0b64414690f0d19bcce2ed78]) successfully deleted
I20240503 14:12:27.101323 30142 log.cc:1196] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/3d137ef4cfc2476cbc70f1cc72ad89e6
I20240503 14:12:27.102133 30142 ts_tablet_manager.cc:1909] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.103159 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 946) took 1358 ms. Trace:
I20240503 14:12:27.104758 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.107108 28212 raft_consensus.cc:3037] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.106992 30142 rpcz_store.cc:276] 0503 14:12:25.744593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.744692 (+ 99us) service_pool.cc:225] Handling call
0503 14:12:27.017769 (+1273077us) ts_tablet_manager.cc:1120] Deleting tablet 3d137ef4cfc2476cbc70f1cc72ad89e6
0503 14:12:27.017784 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.071587 (+ 53803us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.099328 (+ 27741us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.103123 (+ 3795us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23679,"tablet-delete.queue_time_us":1272818}
I20240503 14:12:27.107996 30142 tablet_replica.cc:330] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.107597 30271 consensus_queue.cc:1035] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.109459 30142 raft_consensus.cc:2229] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.109720 30142 raft_consensus.cc:2258] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.110018 28212 raft_consensus.cc:2454] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.110576 28427 leader_election.cc:302] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.000715 28614 tablet_bootstrap.cc:492] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:27.111817 28614 ts_tablet_manager.cc:1367] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.340s user 0.003s sys 0.000s
I20240503 14:12:27.111951 30134 raft_consensus.cc:2781] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.112125 30134 raft_consensus.cc:686] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:27.112386 30134 consensus_queue.cc:237] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.112671 28614 raft_consensus.cc:348] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:27.112869 28614 raft_consensus.cc:374] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.112923 28614 raft_consensus.cc:729] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.113124 28614 consensus_queue.cc:260] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:27.113689 28614 ts_tablet_manager.cc:1398] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:27.113991 28614 tablet_bootstrap.cc:492] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:27.114984 28600 tablet_bootstrap.cc:492] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.115128 28600 ts_tablet_manager.cc:1367] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.003s
I20240503 14:12:27.115634 28614 tablet_bootstrap.cc:654] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.032872 30177 raft_consensus.cc:482] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.116261 30177 raft_consensus.cc:504] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.116742 30142 ts_tablet_manager.cc:1875] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.116998 30177 leader_election.cc:288] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.115991 28600 raft_consensus.cc:348] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.119922 28600 raft_consensus.cc:374] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.120055 28600 raft_consensus.cc:729] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.120316 28600 consensus_queue.cc:260] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.120908 28600 ts_tablet_manager.cc:1398] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:27.121222 28600 tablet_bootstrap.cc:492] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.122974 28600 tablet_bootstrap.cc:654] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.126693 28604 tablet_bootstrap.cc:492] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.126978 28604 ts_tablet_manager.cc:1367] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.004s
I20240503 14:12:27.127851 28604 raft_consensus.cc:348] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.128499 28604 raft_consensus.cc:374] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.128633 28604 raft_consensus.cc:729] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.128973 28604 consensus_queue.cc:260] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:27.087366 28290 consensus_peers.cc:485] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4084bd4fbf604042bb33ff03c3f8cc19. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.130481 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.130848 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.131321 30263 consensus_queue.cc:1035] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.131695 28604 ts_tablet_manager.cc:1398] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:12:27.133515 30211 consensus_queue.cc:1035] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.134325 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
W20240503 14:12:27.134460 28427 leader_election.cc:341] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.134946 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:27.135160 28488 raft_consensus.cc:2454] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
W20240503 14:12:27.135423 28427 leader_election.cc:341] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.138775 30227 tablet.cc:1722] T b75036cdf9e04ea4aa767fd12016f177 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:27.141090 28604 tablet_bootstrap.cc:492] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.145393 29828 log.cc:1196] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/3cbcae22ea124edaa14c9409f3052714
I20240503 14:12:27.145848 29828 ts_tablet_manager.cc:1909] T 3cbcae22ea124edaa14c9409f3052714 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:27.146258 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 965) took 1178 ms. Trace:
I20240503 14:12:27.146356 29828 rpcz_store.cc:276] 0503 14:12:25.967288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.972140 (+ 4852us) service_pool.cc:225] Handling call
0503 14:12:26.811920 (+839780us) ts_tablet_manager.cc:1120] Deleting tablet 3cbcae22ea124edaa14c9409f3052714
0503 14:12:26.811930 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:26.910413 (+ 98483us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:26.999900 (+ 89487us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.146219 (+146319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":839553}
I20240503 14:12:27.146823 29828 tablet_replica.cc:330] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:27.147022 29828 raft_consensus.cc:2229] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.147291 29828 raft_consensus.cc:2258] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.148046 29996 consensus_queue.cc:1035] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.151531 28600 tablet_bootstrap.cc:492] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.151801 28600 ts_tablet_manager.cc:1367] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.004s
I20240503 14:12:27.151819 28604 tablet_bootstrap.cc:654] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.152839 28600 raft_consensus.cc:348] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.153163 28600 raft_consensus.cc:374] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.153281 28600 raft_consensus.cc:729] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.153744 28600 consensus_queue.cc:260] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.154438 28600 ts_tablet_manager.cc:1398] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:27.154819 28600 tablet_bootstrap.cc:492] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.160243 28600 tablet_bootstrap.cc:654] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.164798 28661 ts_tablet_manager.cc:1888] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.171366 28614 tablet_bootstrap.cc:492] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:27.171656 28614 ts_tablet_manager.cc:1367] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.002s
I20240503 14:12:27.171653 30142 ts_tablet_manager.cc:1888] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.171785 30142 log.cc:1196] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/57ab077988364bd687e54f90c3c265d5
W20240503 14:12:27.174252 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 799eed37c032486ab5479cfdb6d5a045 consensus state has no leader
I20240503 14:12:27.174849 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e3b970f0246245259885e419a838a1a9" } new_table_name: "default.Table_707d25a0650a44c1abc15a601d6037c0" modify_external_catalogs: true
I20240503 14:12:27.175297 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0e114c978c714945b50f17d008fb60d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.175925 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e114c978c714945b50f17d008fb60d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.176564 28661 log.cc:1196] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/920be6d87c4440d0bf7453c86cfe0b5c
I20240503 14:12:27.177086 28661 ts_tablet_manager.cc:1909] T 920be6d87c4440d0bf7453c86cfe0b5c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.177373 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e04c0b6955b34267820ceecfffcfb88a
I20240503 14:12:27.178299 28027 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
I20240503 14:12:27.178231 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_346e6ce2bc2b4524a52ec6ac63cbe9d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.178810 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_346e6ce2bc2b4524a52ec6ac63cbe9d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.178607 28027 catalog_manager.cc:5484] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:27.179092 28027 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:27.179335 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 975) took 1441 ms. Trace:
I20240503 14:12:27.178165 28604 tablet_bootstrap.cc:492] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.179795 28604 ts_tablet_manager.cc:1367] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.001s
I20240503 14:12:27.180037 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a526e9f1e9934e519349dcd4523ee694
I20240503 14:12:27.180716 28604 raft_consensus.cc:348] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.180847 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e6e71c3cb9d7443c8354b100cd955068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:27.180922 28604 raft_consensus.cc:374] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.181363 28604 raft_consensus.cc:729] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:27.181432 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6e71c3cb9d7443c8354b100cd955068 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.181596 28604 consensus_queue.cc:260] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.182634 28604 ts_tablet_manager.cc:1398] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:27.182940 28604 tablet_bootstrap.cc:492] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.184474 29996 raft_consensus.cc:482] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.184648 28604 tablet_bootstrap.cc:654] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.184630 29996 raft_consensus.cc:504] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.179423 28661 rpcz_store.cc:276] 0503 14:12:25.737391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.737472 (+ 81us) service_pool.cc:225] Handling call
0503 14:12:27.098421 (+1360949us) ts_tablet_manager.cc:1120] Deleting tablet 920be6d87c4440d0bf7453c86cfe0b5c
0503 14:12:27.098434 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.161448 (+ 63014us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.164648 (+ 3200us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.179309 (+ 14661us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360728}
I20240503 14:12:27.185093 28661 tablet_replica.cc:330] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.185308 29996 leader_election.cc:288] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.185302 28661 raft_consensus.cc:2229] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.185587 28661 raft_consensus.cc:2258] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.186970 28661 ts_tablet_manager.cc:1875] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.186367 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884cf4ea11e48ef941722b79ad50075" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.187983 28212 raft_consensus.cc:2454] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.188593 28031 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 6a1212dcb62347f1a28e621b97ca349a
I20240503 14:12:27.188597 28427 leader_election.cc:302] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.188856 29996 raft_consensus.cc:2781] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.188925 29996 raft_consensus.cc:482] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.188980 29996 raft_consensus.cc:3037] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.189079 30142 ts_tablet_manager.cc:1909] T 57ab077988364bd687e54f90c3c265d5 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.189944 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 947) took 1444 ms. Trace:
I20240503 14:12:27.190032 30142 rpcz_store.cc:276] 0503 14:12:25.745394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.745451 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:27.107939 (+1362488us) ts_tablet_manager.cc:1120] Deleting tablet 57ab077988364bd687e54f90c3c265d5
0503 14:12:27.107951 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.127582 (+ 19631us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.171515 (+ 43933us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.189919 (+ 18404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362290}
I20240503 14:12:27.191581 29996 raft_consensus.cc:504] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.192304 29996 leader_election.cc:288] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.193087 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884cf4ea11e48ef941722b79ad50075" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.193336 28212 raft_consensus.cc:3037] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.193207 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cd02de883e9a43a79775ffc0e5deb283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:27.193421 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 948) took 1422 ms. Trace:
I20240503 14:12:27.193502 30142 rpcz_store.cc:276] 0503 14:12:25.771166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.771238 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:27.193270 (+1422032us) ts_tablet_manager.cc:1120] Deleting tablet 6a27d21de6a6441d810f379bba88f9a2
0503 14:12:27.193282 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.193399 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1421777}
W20240503 14:12:27.193851 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd02de883e9a43a79775ffc0e5deb283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.196067 28212 raft_consensus.cc:2454] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.197373 28427 leader_election.cc:302] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.197769 29994 raft_consensus.cc:482] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.197907 29994 raft_consensus.cc:504] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:27.198484 29994 leader_election.cc:288] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:27.198618 29994 raft_consensus.cc:2781] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.198714 29994 raft_consensus.cc:686] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:27.198940 29994 consensus_queue.cc:237] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.201711 28614 raft_consensus.cc:348] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.202804 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d54cc5c03034a6685c1da34f9bd65ac" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.203152 28212 raft_consensus.cc:2454] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.203507 30266 raft_consensus.cc:482] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.203862 30266 raft_consensus.cc:504] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.204622 28427 leader_election.cc:302] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.204692 30266 leader_election.cc:288] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.204885 29994 raft_consensus.cc:2781] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.205024 29994 raft_consensus.cc:482] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.205153 29994 raft_consensus.cc:3037] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.207597 29994 raft_consensus.cc:504] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:27.208420 29994 leader_election.cc:288] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:27.209335 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d54cc5c03034a6685c1da34f9bd65ac" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.209674 28212 raft_consensus.cc:3037] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.212461 28212 raft_consensus.cc:2454] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.214222 28427 leader_election.cc:302] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.214550 28600 tablet_bootstrap.cc:492] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.214651 29994 raft_consensus.cc:2781] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.214738 28600 ts_tablet_manager.cc:1367] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.006s
I20240503 14:12:27.214768 29994 raft_consensus.cc:686] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:27.215039 29994 consensus_queue.cc:237] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:27.215595 28600 raft_consensus.cc:348] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.215826 28600 raft_consensus.cc:374] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.215926 28600 raft_consensus.cc:729] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.216193 28600 consensus_queue.cc:260] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.216801 28600 ts_tablet_manager.cc:1398] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:27.217144 28600 tablet_bootstrap.cc:492] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.219727 28600 tablet_bootstrap.cc:654] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.220921 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884cf4ea11e48ef941722b79ad50075" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.223385 28614 raft_consensus.cc:374] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.223567 28614 raft_consensus.cc:729] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.220948 28350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884cf4ea11e48ef941722b79ad50075" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.227733 28661 ts_tablet_manager.cc:1888] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.227823 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet e04c0b6955b34267820ceecfffcfb88a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.227877 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet a526e9f1e9934e519349dcd4523ee694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.196244 30142 tablet_replica.cc:330] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.230580 30142 raft_consensus.cc:2229] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.231635 30142 raft_consensus.cc:2258] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.232980 30142 ts_tablet_manager.cc:1875] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.235569 28614 consensus_queue.cc:260] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.236580 28614 ts_tablet_manager.cc:1398] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20240503 14:12:27.236981 28614 tablet_bootstrap.cc:492] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:27.238613 28614 tablet_bootstrap.cc:654] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.239540 29828 ts_tablet_manager.cc:1875] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.241765 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c7a51c68ef478e9bdc5b3e3dc3d5df" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.245072 28600 tablet_bootstrap.cc:492] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.245296 28600 ts_tablet_manager.cc:1367] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.006s
I20240503 14:12:27.245568 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 920be6d87c4440d0bf7453c86cfe0b5c (table default.Table_d8cb0703edfb4e96ba081bd6f299d867 [id=6119a6b076964647bd7247e0a896a030]) successfully deleted
I20240503 14:12:27.245916 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 (table default.Table_2e88623765c84939b082bcf13612fd61 [id=7ca3beb0b91840d08d178367a46c9936]) successfully deleted
I20240503 14:12:27.246243 28600 raft_consensus.cc:348] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.246490 28600 raft_consensus.cc:374] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.246243 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 57ab077988364bd687e54f90c3c265d5 (table default.Table_cdaf5876f13c4bd3adb7cb8aea0cfc88 [id=7877e3c7f4b14fcebfd7bec2e15b6240]) successfully deleted
I20240503 14:12:27.246578 28600 raft_consensus.cc:729] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
W20240503 14:12:27.246861 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6a27d21de6a6441d810f379bba88f9a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a27d21de6a6441d810f379bba88f9a2 already in progress: creating tablet
I20240503 14:12:27.246845 28600 consensus_queue.cc:260] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.247427 28600 ts_tablet_manager.cc:1398] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:27.247812 28600 tablet_bootstrap.cc:492] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.229084 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet e04c0b6955b34267820ceecfffcfb88a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.248747 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d54cc5c03034a6685c1da34f9bd65ac" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.248952 28661 log.cc:1196] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/3cbcae22ea124edaa14c9409f3052714
I20240503 14:12:27.229260 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet a526e9f1e9934e519349dcd4523ee694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.248984 28355 raft_consensus.cc:2454] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.247450 30142 ts_tablet_manager.cc:1888] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.249256 30142 log.cc:1196] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ff1958dc488e4854a8c8e74da9fab243
I20240503 14:12:27.249440 28661 ts_tablet_manager.cc:1909] T 3cbcae22ea124edaa14c9409f3052714 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.249724 28600 tablet_bootstrap.cc:654] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.249667 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d54cc5c03034a6685c1da34f9bd65ac" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.249840 28355 raft_consensus.cc:3037] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.236807 28604 tablet_bootstrap.cc:492] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.251005 28604 ts_tablet_manager.cc:1367] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.004s
I20240503 14:12:27.251778 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 976) took 1514 ms. Trace:
I20240503 14:12:27.251909 28661 rpcz_store.cc:276] 0503 14:12:25.737507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.737559 (+ 52us) service_pool.cc:225] Handling call
0503 14:12:27.185034 (+1447475us) ts_tablet_manager.cc:1120] Deleting tablet 3cbcae22ea124edaa14c9409f3052714
0503 14:12:27.185047 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.224063 (+ 39016us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.227578 (+ 3515us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.251701 (+ 24123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1447280}
I20240503 14:12:27.252051 28604 raft_consensus.cc:348] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.252332 28604 raft_consensus.cc:374] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.252418 28604 raft_consensus.cc:729] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.252728 28604 consensus_queue.cc:260] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.253278 28604 ts_tablet_manager.cc:1398] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:27.253602 28355 raft_consensus.cc:2454] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.253640 28604 tablet_bootstrap.cc:492] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.255358 28661 tablet_replica.cc:330] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.255563 28661 raft_consensus.cc:2229] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.255760 28661 raft_consensus.cc:2258] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.257337 28661 ts_tablet_manager.cc:1875] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.262780 28604 tablet_bootstrap.cc:654] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.264513 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c7a51c68ef478e9bdc5b3e3dc3d5df" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:27.273649 30263 raft_consensus.cc:482] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.273847 30263 raft_consensus.cc:504] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } }
I20240503 14:12:27.274559 30263 leader_election.cc:288] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:27.276923 28427 leader_election.cc:341] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:27.277612 28427 leader_election.cc:341] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.278831 30263 raft_consensus.cc:482] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.278973 30263 raft_consensus.cc:504] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.279028 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50b3e199314bfc8465fdb7b2425d07" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.279583 30263 leader_election.cc:288] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.280267 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50b3e199314bfc8465fdb7b2425d07" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:27.280622 28486 raft_consensus.cc:2379] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
W20240503 14:12:27.288327 28150 leader_election.cc:341] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.294862 28661 ts_tablet_manager.cc:1888] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:27.301368 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 409cbf0cb8ef44819cfdb36f2fe83110 consensus state has no leader
I20240503 14:12:27.301868 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_af7c6d0b47e24c60b73741ec9e1f6d7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:27.302440 28024 master_service.cc:946] ReplaceTablet: received request to replace tablet b97da778da214bb6bc147c5ada6c7603 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:27.302500 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af7c6d0b47e24c60b73741ec9e1f6d7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.303136 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_981c33c4c24e4e848a74bde5c0a875d7" } new_table_name: "default.Table_0a9fb4d2c7da4d4fb2ceaa93e5db65f3" modify_external_catalogs: true
I20240503 14:12:27.303700 28028 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_87bcc8d1e5244ebdb6ed58c62b34f709 [id=30ff854aabe84b2cb03824d287ae5685] alter complete (version 1)
I20240503 14:12:27.304567 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4eec071da1eb4980b74b08ce5b21765b" } modify_external_catalogs: true
I20240503 14:12:27.304749 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4eec071da1eb4980b74b08ce5b21765b" } modify_external_catalogs: true
I20240503 14:12:27.306005 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_77a20b54283b43be95a5e60a0a9dbd05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.306569 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77a20b54283b43be95a5e60a0a9dbd05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.308176 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_47f02eacca0d41e3a4d9913ed4768151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.308815 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47f02eacca0d41e3a4d9913ed4768151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.314527 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ab077988364bd687e54f90c3c265d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cd037e135904e2cab09cbf39fed0b13) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.316015 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.306567 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b517009a45264b2eb9d5c01512efabc3" } new_table_name: "default.Table_a6fe66b4dbee47e9b0b15b491131bfe7" modify_external_catalogs: true
I20240503 14:12:27.321774 30272 raft_consensus.cc:482] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.322065 30272 raft_consensus.cc:504] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.322835 30272 leader_election.cc:288] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:27.323204 28150 leader_election.cc:341] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.323387 28150 leader_election.cc:302] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.323661 30272 raft_consensus.cc:2726] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.326346 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.326597 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet a526e9f1e9934e519349dcd4523ee694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.326597 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet e04c0b6955b34267820ceecfffcfb88a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.327001 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50b3e199314bfc8465fdb7b2425d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.327873 28661 log.cc:1196] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/8fe3affd0d144b1fb0078e7690b9b36d
I20240503 14:12:27.315258 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.315570 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.328459 28661 ts_tablet_manager.cc:1909] T 8fe3affd0d144b1fb0078e7690b9b36d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.330832 28290 leader_election.cc:302] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:27.331154 30263 raft_consensus.cc:2781] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:27.331229 30263 raft_consensus.cc:482] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a64bc15d65d84ad5a005960a9a84df9c)
I20240503 14:12:27.331305 30263 raft_consensus.cc:3037] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:27.331410 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 977) took 1593 ms. Trace:
I20240503 14:12:27.331537 28661 rpcz_store.cc:276] 0503 14:12:25.737600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.737727 (+ 127us) service_pool.cc:225] Handling call
0503 14:12:27.254626 (+1516899us) ts_tablet_manager.cc:1120] Deleting tablet 8fe3affd0d144b1fb0078e7690b9b36d
0503 14:12:27.254636 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.282487 (+ 27851us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.294654 (+ 12167us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.331364 (+ 36710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516766}
I20240503 14:12:27.331894 28661 tablet_replica.cc:330] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.332134 28661 raft_consensus.cc:2229] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.332335 28661 raft_consensus.cc:2258] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.335110 30263 raft_consensus.cc:504] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.336051 30263 leader_election.cc:288] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.343307 30272 raft_consensus.cc:482] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.343591 30272 raft_consensus.cc:504] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.344349 30272 leader_election.cc:288] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.344995 29828 ts_tablet_manager.cc:1888] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.345584 30142 ts_tablet_manager.cc:1909] T ff1958dc488e4854a8c8e74da9fab243 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.346820 28661 ts_tablet_manager.cc:1875] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.349014 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 956) took 1558 ms. Trace:
I20240503 14:12:27.349254 30142 rpcz_store.cc:276] 0503 14:12:25.790180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.790230 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:27.196161 (+1405931us) ts_tablet_manager.cc:1120] Deleting tablet ff1958dc488e4854a8c8e74da9fab243
0503 14:12:27.196173 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.236575 (+ 40402us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.247291 (+ 10716us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.348984 (+101693us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":483,"tablet-delete.queue_time_us":1405677}
I20240503 14:12:27.350522 30142 tablet_replica.cc:330] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.320056 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 3cbcae22ea124edaa14c9409f3052714 (table default.Table_6d7202481c4747ddb44a78ba8bb0063f [id=bafd499c743040d08a2fd8207d4f1e2e]) successfully deleted
I20240503 14:12:27.351940 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 3cbcae22ea124edaa14c9409f3052714 (table default.Table_6d7202481c4747ddb44a78ba8bb0063f [id=bafd499c743040d08a2fd8207d4f1e2e]) successfully deleted
I20240503 14:12:27.352183 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 8fe3affd0d144b1fb0078e7690b9b36d (table default.Table_32b3856710f446139fa555cd42254eef [id=cf15c50bd96d4420b98a06df477003b6]) successfully deleted
I20240503 14:12:27.352699 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ff1958dc488e4854a8c8e74da9fab243 (table default.Table_8ea2f91bb9e242aeb07b426cd711d537 [id=aed23d17e9704c36b4b4ecb604987e37]) successfully deleted
I20240503 14:12:27.353428 30142 raft_consensus.cc:2229] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.353863 30142 raft_consensus.cc:2258] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.357707 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_032d8e14bf694d2fbcd3b37c7ef106f9" } modify_external_catalogs: true
I20240503 14:12:27.358011 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_032d8e14bf694d2fbcd3b37c7ef106f9" } modify_external_catalogs: true
W20240503 14:12:27.358347 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b46cfed6c8e42a6acf4d1fb128a8656 consensus state has no leader
I20240503 14:12:27.358805 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_45c301ad86394025a523ce52461c5606"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.359414 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45c301ad86394025a523ce52461c5606 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.360638 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7aa3d349d8e24199ada0764815b8d96e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.361152 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aa3d349d8e24199ada0764815b8d96e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.367193 30142 ts_tablet_manager.cc:1875] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.368266 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet b97da778da214bb6bc147c5ada6c7603
I20240503 14:12:27.368903 28024 catalog_manager.cc:6554] ReplaceTablet: tablet b97da778da214bb6bc147c5ada6c7603 deleted and replaced by tablet cc815484ff24435fab5d994e4e564668
W20240503 14:12:27.369508 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6518f93f7a5c444aa91c9473a5111e5b has no consensus state
I20240503 14:12:27.373349 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f91ddbc250c44fbfbf4ffe83433d8dd8
I20240503 14:12:27.379279 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6bfb8beb36ba4f4292703a3b2224d553" } modify_external_catalogs: true
I20240503 14:12:27.379554 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6bfb8beb36ba4f4292703a3b2224d553" } modify_external_catalogs: true
I20240503 14:12:27.380232 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_e98a15184c3d4811aedd7a0ac8d4fb9d" } new_table_name: "default.Table_be0416fa4db54ee0ad44f2faaad25569" modify_external_catalogs: true
I20240503 14:12:27.381949 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d8f7551fc7ad4ea28e5a730ab6f2d0d4" } modify_external_catalogs: true
I20240503 14:12:27.382190 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d8f7551fc7ad4ea28e5a730ab6f2d0d4" } modify_external_catalogs: true
I20240503 14:12:27.382761 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0a2e433bdb54478588ee6b8965e406fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.383561 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a2e433bdb54478588ee6b8965e406fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.387459 28600 tablet_bootstrap.cc:492] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.387692 28600 ts_tablet_manager.cc:1367] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.140s user 0.038s sys 0.030s
I20240503 14:12:27.388597 28600 raft_consensus.cc:348] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.388965 28600 raft_consensus.cc:374] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.389079 28600 raft_consensus.cc:729] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.389328 28661 ts_tablet_manager.cc:1888] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.389403 28600 consensus_queue.cc:260] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.390080 28600 ts_tablet_manager.cc:1398] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:27.390439 28600 tablet_bootstrap.cc:492] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.391865 28600 tablet_bootstrap.cc:654] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.394522 28661 log.cc:1196] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ff1958dc488e4854a8c8e74da9fab243
I20240503 14:12:27.395071 28661 ts_tablet_manager.cc:1909] T ff1958dc488e4854a8c8e74da9fab243 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.393842 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e4f6e3908e420e9fa156ee24b437d6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:27.401477 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 979) took 1663 ms. Trace:
I20240503 14:12:27.401621 28661 rpcz_store.cc:276] 0503 14:12:25.737796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.737835 (+ 39us) service_pool.cc:225] Handling call
0503 14:12:27.331823 (+1593988us) ts_tablet_manager.cc:1120] Deleting tablet ff1958dc488e4854a8c8e74da9fab243
0503 14:12:27.331834 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.373095 (+ 41261us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.389176 (+ 16081us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.401407 (+ 12231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593846}
I20240503 14:12:27.402053 28661 tablet_replica.cc:330] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.402289 28661 raft_consensus.cc:2229] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.402441 30142 ts_tablet_manager.cc:1888] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.402520 28661 raft_consensus.cc:2258] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.402670 30142 log.cc:1196] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/d93fea5caec94695a5df26d3630c50c7
I20240503 14:12:27.403141 30142 ts_tablet_manager.cc:1909] T d93fea5caec94695a5df26d3630c50c7 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.403151 28661 ts_tablet_manager.cc:1875] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.404124 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 957) took 1613 ms. Trace:
I20240503 14:12:27.404268 30142 rpcz_store.cc:276] 0503 14:12:25.790810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.790858 (+ 48us) service_pool.cc:225] Handling call
0503 14:12:27.350451 (+1559593us) ts_tablet_manager.cc:1120] Deleting tablet d93fea5caec94695a5df26d3630c50c7
0503 14:12:27.350467 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.377057 (+ 26590us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.402236 (+ 25179us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.404099 (+ 1863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559412}
I20240503 14:12:27.404953 30142 tablet_replica.cc:330] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.405198 30142 raft_consensus.cc:2229] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.405589 30142 raft_consensus.cc:2258] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.406270 30142 ts_tablet_manager.cc:1875] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.406692 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b41e3b21347b393497de4d4e87469" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:27.407063 28486 raft_consensus.cc:2454] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
W20240503 14:12:27.410890 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 44984.8us
I20240503 14:12:27.411656 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet b97da778da214bb6bc147c5ada6c7603 with delete_type TABLET_DATA_DELETED (replaced by tablet cc815484ff24435fab5d994e4e564668) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.415318 29994 raft_consensus.cc:482] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:27.415305 28150 leader_election.cc:341] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.415529 28150 leader_election.cc:302] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.415484 29994 raft_consensus.cc:504] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.415758 30272 raft_consensus.cc:2781] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.415825 30272 raft_consensus.cc:482] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.415879 30272 raft_consensus.cc:3037] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.416256 29994 leader_election.cc:288] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.417198 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4846d662a64d51b16fd5447c371e9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.417402 28215 raft_consensus.cc:2454] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.418002 28427 leader_election.cc:302] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.418452 29994 raft_consensus.cc:2781] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.418432 30272 raft_consensus.cc:504] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.418570 29994 raft_consensus.cc:482] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.418658 29994 raft_consensus.cc:3037] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.419076 30272 leader_election.cc:288] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.420351 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet b97da778da214bb6bc147c5ada6c7603 with delete_type TABLET_DATA_DELETED (replaced by tablet cc815484ff24435fab5d994e4e564668) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.420737 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.421295 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.422143 29994 raft_consensus.cc:504] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.423609 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4846d662a64d51b16fd5447c371e9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.423821 28215 raft_consensus.cc:3037] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.426394 28215 raft_consensus.cc:2454] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.427186 28427 leader_election.cc:302] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.427884 29996 raft_consensus.cc:2781] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.428066 29996 raft_consensus.cc:686] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:27.428375 29996 consensus_queue.cc:237] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.428848 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet d93fea5caec94695a5df26d3630c50c7 (table default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]) successfully deleted
I20240503 14:12:27.429148 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ff1958dc488e4854a8c8e74da9fab243 (table default.Table_8ea2f91bb9e242aeb07b426cd711d537 [id=aed23d17e9704c36b4b4ecb604987e37]) successfully deleted
I20240503 14:12:27.431152 30142 ts_tablet_manager.cc:1888] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.431454 30142 log.cc:1196] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/dcd405fd84d44076b1792b7fcbe00e84
I20240503 14:12:27.431862 30142 ts_tablet_manager.cc:1909] T dcd405fd84d44076b1792b7fcbe00e84 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.432746 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 953) took 1643 ms. Trace:
I20240503 14:12:27.432929 30142 rpcz_store.cc:276] 0503 14:12:25.789475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.800147 (+ 10672us) service_pool.cc:225] Handling call
0503 14:12:27.404899 (+1604752us) ts_tablet_manager.cc:1120] Deleting tablet dcd405fd84d44076b1792b7fcbe00e84
0503 14:12:27.404910 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.416534 (+ 11624us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.431049 (+ 14515us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.432716 (+ 1667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1604560}
I20240503 14:12:27.433468 30142 tablet_replica.cc:330] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.431739 29994 leader_election.cc:288] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.433717 30142 raft_consensus.cc:2229] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.430140 28661 ts_tablet_manager.cc:1888] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.433882 30142 raft_consensus.cc:2258] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.434080 28661 log.cc:1196] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/4fad4508a29d4822b81af6947b57e7ef
I20240503 14:12:27.434470 30142 ts_tablet_manager.cc:1875] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.436852 28661 ts_tablet_manager.cc:1909] T 4fad4508a29d4822b81af6947b57e7ef P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.433157 28600 tablet_bootstrap.cc:492] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.439664 28600 ts_tablet_manager.cc:1367] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.049s user 0.021s sys 0.002s
I20240503 14:12:27.440670 28600 raft_consensus.cc:348] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.440917 28600 raft_consensus.cc:374] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.441002 28600 raft_consensus.cc:729] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.440958 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b41e3b21347b393497de4d4e87469" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:27.441258 28486 raft_consensus.cc:3037] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.441256 28600 consensus_queue.cc:260] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.441771 28600 ts_tablet_manager.cc:1398] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:27.442055 28600 tablet_bootstrap.cc:492] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.443430 28600 tablet_bootstrap.cc:654] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.445107 28604 tablet_bootstrap.cc:492] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.445310 28604 ts_tablet_manager.cc:1367] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.192s user 0.026s sys 0.045s
I20240503 14:12:27.446180 28486 raft_consensus.cc:2454] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:27.393013 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e4f6e3908e420e9fa156ee24b437d6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.446350 29828 log.cc:1196] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/80cd09e769be4bee95bbef9c3c550451
I20240503 14:12:27.446705 29828 ts_tablet_manager.cc:1909] T 80cd09e769be4bee95bbef9c3c550451 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:27.394338 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 63544.3us
I20240503 14:12:27.397955 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b41e3b21347b393497de4d4e87469" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.447098 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 966) took 1474 ms. Trace:
I20240503 14:12:27.447187 29828 rpcz_store.cc:276] 0503 14:12:25.972554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.973317 (+ 763us) service_pool.cc:225] Handling call
0503 14:12:27.146763 (+1173446us) ts_tablet_manager.cc:1120] Deleting tablet 80cd09e769be4bee95bbef9c3c550451
0503 14:12:27.146775 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.341519 (+194744us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.344874 (+ 3355us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.447077 (+102203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1173257}
I20240503 14:12:27.447503 29828 tablet_replica.cc:330] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:27.447705 29828 raft_consensus.cc:2229] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.447882 29828 raft_consensus.cc:2258] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.448243 30263 consensus_queue.cc:1035] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.448411 30142 ts_tablet_manager.cc:1888] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.448545 30142 log.cc:1196] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/20b176a071ba4f3fbff2a64d6e46027a
I20240503 14:12:27.431573 28614 tablet_bootstrap.cc:492] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:27.435519 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b41e3b21347b393497de4d4e87469" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.448848 30142 ts_tablet_manager.cc:1909] T 20b176a071ba4f3fbff2a64d6e46027a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.449082 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet dcd405fd84d44076b1792b7fcbe00e84 (table default.Table_92abfaed91ab4683a6b2e501ce673e58 [id=6539aa29cb864a2fa19c8a2f1ac7bd19]) successfully deleted
I20240503 14:12:27.450822 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 954) took 1661 ms. Trace:
I20240503 14:12:27.450911 30142 rpcz_store.cc:276] 0503 14:12:25.789696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.800379 (+ 10683us) service_pool.cc:225] Handling call
0503 14:12:27.433409 (+1633030us) ts_tablet_manager.cc:1120] Deleting tablet 20b176a071ba4f3fbff2a64d6e46027a
0503 14:12:27.433420 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.437377 (+ 3957us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.445822 (+ 8445us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.450793 (+ 4971us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1632895}
I20240503 14:12:27.451233 30142 tablet_replica.cc:330] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.452322 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8ae143ed239d4c81bd80ed397373c7da" } modify_external_catalogs: true
I20240503 14:12:27.452541 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8ae143ed239d4c81bd80ed397373c7da" } modify_external_catalogs: true
I20240503 14:12:27.457603 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.460705 30142 raft_consensus.cc:2229] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.460979 30142 raft_consensus.cc:2258] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.462905 30142 ts_tablet_manager.cc:1875] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.464367 28614 ts_tablet_manager.cc:1367] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.227s user 0.002s sys 0.002s
I20240503 14:12:27.465237 28614 raft_consensus.cc:348] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.465484 28614 raft_consensus.cc:374] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.465541 28614 raft_consensus.cc:729] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.466665 28215 raft_consensus.cc:1261] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Refusing update from remote peer d3a6b9b7913444d29c96581af6febf33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:27.470441 28614 consensus_queue.cc:260] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.446074 28604 raft_consensus.cc:348] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.460363 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:27.471398 28604 raft_consensus.cc:374] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.471490 28604 raft_consensus.cc:729] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.471686 28604 consensus_queue.cc:260] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.472276 28604 ts_tablet_manager.cc:1398] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240503 14:12:27.472597 28604 tablet_bootstrap.cc:492] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.474236 28604 tablet_bootstrap.cc:654] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.471016 28290 leader_election.cc:302] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.475731 28028 catalog_manager.cc:5484] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:27.476688 30263 raft_consensus.cc:3037] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.479629 30263 raft_consensus.cc:2726] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.448632 29828 ts_tablet_manager.cc:1875] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.483844 30142 ts_tablet_manager.cc:1888] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:27.488173 28290 leader_election.cc:341] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 7b6c998e518541248a87e4c1551ec080
I20240503 14:12:27.488878 30142 log.cc:1196] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/4e8411f474c247c2abfd0bd9682fb8a2
I20240503 14:12:27.489529 30142 ts_tablet_manager.cc:1909] T 4e8411f474c247c2abfd0bd9682fb8a2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.490947 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 955) took 1701 ms. Trace:
I20240503 14:12:27.492143 30142 rpcz_store.cc:276] 0503 14:12:25.789945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.800537 (+ 10592us) service_pool.cc:225] Handling call
0503 14:12:27.451180 (+1650643us) ts_tablet_manager.cc:1120] Deleting tablet 4e8411f474c247c2abfd0bd9682fb8a2
0503 14:12:27.451191 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.469551 (+ 18360us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.483712 (+ 14161us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.490916 (+ 7204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1650527}
I20240503 14:12:27.493338 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 960) took 1498 ms. Trace:
I20240503 14:12:27.493856 28614 ts_tablet_manager.cc:1398] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240503 14:12:27.493960 30142 rpcz_store.cc:276] 0503 14:12:25.994555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.996817 (+ 2262us) service_pool.cc:225] Handling call
0503 14:12:27.493161 (+1496344us) ts_tablet_manager.cc:1120] Deleting tablet 5de9edfb0d064939bda2ec7a9e0e500b
0503 14:12:27.493175 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.493306 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491995}
I20240503 14:12:27.494329 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 20b176a071ba4f3fbff2a64d6e46027a (table default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]) successfully deleted
I20240503 14:12:27.494629 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 4e8411f474c247c2abfd0bd9682fb8a2 (table default.Table_fc189ad8c19041f7814fac6de9f482cb [id=b7f4f49135e245dc975ffd7eb0589a8d]) successfully deleted
I20240503 14:12:27.494860 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 80cd09e769be4bee95bbef9c3c550451 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
I20240503 14:12:27.495110 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 961) took 1363 ms. Trace:
I20240503 14:12:27.495374 30142 rpcz_store.cc:276] 0503 14:12:26.131869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.131945 (+ 76us) service_pool.cc:225] Handling call
0503 14:12:27.494703 (+1362758us) ts_tablet_manager.cc:1120] Deleting tablet 0d4c5648ddc84407a8d8b44f2d63dcee
0503 14:12:27.494888 (+ 185us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.495091 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362487}
I20240503 14:12:27.496014 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 962) took 1339 ms. Trace:
I20240503 14:12:27.499296 28614 tablet_bootstrap.cc:492] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:27.496378 30272 raft_consensus.cc:482] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.499744 30272 raft_consensus.cc:504] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.500480 30272 leader_election.cc:288] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.500854 30142 rpcz_store.cc:276] 0503 14:12:26.156548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.156610 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:27.495908 (+1339298us) ts_tablet_manager.cc:1120] Deleting tablet 19d43cc864064c4c907ec0629391da10
0503 14:12:27.495917 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.496000 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339046}
I20240503 14:12:27.503057 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 963) took 1345 ms. Trace:
I20240503 14:12:27.503775 30142 rpcz_store.cc:276] 0503 14:12:26.157538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.157594 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:27.502876 (+1345282us) ts_tablet_manager.cc:1120] Deleting tablet 13f8553cfbab4af2a38a73e81fed2a5f
0503 14:12:27.502891 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.503031 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1345026}
I20240503 14:12:27.504601 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 964) took 1346 ms. Trace:
I20240503 14:12:27.505187 30142 rpcz_store.cc:276] 0503 14:12:26.158195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.158255 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:27.504439 (+1346184us) ts_tablet_manager.cc:1120] Deleting tablet 9f5b973ea16d42babd2cc75e32624854
0503 14:12:27.504452 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.504577 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1345980}
I20240503 14:12:27.508276 30142 tablet_replica.cc:330] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.508949 30142 raft_consensus.cc:2229] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.509821 30142 raft_consensus.cc:2258] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.511344 30142 ts_tablet_manager.cc:1875] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.513706 30263 consensus_queue.cc:1035] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.515918 28150 leader_election.cc:302] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.516155 28604 tablet_bootstrap.cc:492] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.516330 28604 ts_tablet_manager.cc:1367] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
W20240503 14:12:27.521057 28019 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 409cbf0cb8ef44819cfdb36f2fe83110 (table default.Table_707d25a0650a44c1abc15a601d6037c0 [id=fac1d13c6016463eba476f16c80ed7e5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 409cbf0cb8ef44819cfdb36f2fe83110 consensus state has no leader
I20240503 14:12:27.521672 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 978) took 1783 ms. Trace:
I20240503 14:12:27.521771 28661 rpcz_store.cc:276] 0503 14:12:25.737698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.737889 (+ 191us) service_pool.cc:225] Handling call
0503 14:12:27.401946 (+1664057us) ts_tablet_manager.cc:1120] Deleting tablet 4fad4508a29d4822b81af6947b57e7ef
0503 14:12:27.401957 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.406161 (+ 4204us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.429974 (+ 23813us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.521638 (+ 91664us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663934}
I20240503 14:12:27.522150 30142 ts_tablet_manager.cc:1888] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.522224 28661 tablet_replica.cc:330] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.522270 30142 log.cc:1196] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/11ac912a3bd64cdc9082596e09c72e13
I20240503 14:12:27.522439 28661 raft_consensus.cc:2229] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.522609 30142 ts_tablet_manager.cc:1909] T 11ac912a3bd64cdc9082596e09c72e13 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.522611 28661 raft_consensus.cc:2258] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.523221 28661 ts_tablet_manager.cc:1875] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.518242 28604 raft_consensus.cc:348] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.525723 28604 raft_consensus.cc:374] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.525619 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2300b3e81cff4e629ac39007ac3ba4af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:27.525791 28604 raft_consensus.cc:729] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
W20240503 14:12:27.526295 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2300b3e81cff4e629ac39007ac3ba4af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.527740 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_463c966bd4c847d0a0967bc7dae04ec6" } new_table_name: "default.Table_7d85d0d3a24348cbbed0f62ccae20a77" modify_external_catalogs: true
I20240503 14:12:27.530948 28600 tablet_bootstrap.cc:492] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.531127 28600 ts_tablet_manager.cc:1367] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.009s
I20240503 14:12:27.531857 28604 consensus_queue.cc:260] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.536662 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4846d662a64d51b16fd5447c371e9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.536935 28604 ts_tablet_manager.cc:1398] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240503 14:12:27.537453 30272 raft_consensus.cc:2781] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.537606 30272 raft_consensus.cc:686] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:27.537940 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 965) took 1331 ms. Trace:
I20240503 14:12:27.537914 30272 consensus_queue.cc:237] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.538031 30142 rpcz_store.cc:276] 0503 14:12:26.206140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.206201 (+ 61us) service_pool.cc:225] Handling call
0503 14:12:27.508209 (+1302008us) ts_tablet_manager.cc:1120] Deleting tablet 11ac912a3bd64cdc9082596e09c72e13
0503 14:12:27.508222 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.519056 (+ 10834us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.522027 (+ 2971us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.537912 (+ 15885us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14393,"tablet-delete.queue_time_us":1301724}
I20240503 14:12:27.538735 30142 tablet_replica.cc:330] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.538938 30142 raft_consensus.cc:2229] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.539150 30142 raft_consensus.cc:2258] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.539330 28600 raft_consensus.cc:348] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.539554 28600 raft_consensus.cc:374] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.539623 28600 raft_consensus.cc:729] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.539845 28600 consensus_queue.cc:260] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.540472 30142 ts_tablet_manager.cc:1875] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.543926 29828 ts_tablet_manager.cc:1888] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.540475 28600 ts_tablet_manager.cc:1398] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:12:27.545145 28600 tablet_bootstrap.cc:492] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.546669 28600 tablet_bootstrap.cc:654] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.547657 29994 raft_consensus.cc:482] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.547827 29994 raft_consensus.cc:504] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.548497 29994 leader_election.cc:288] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.549352 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.549595 28215 raft_consensus.cc:2379] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 1: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:27.552188 28604 tablet_bootstrap.cc:492] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.553242 30142 ts_tablet_manager.cc:1888] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.553494 30142 log.cc:1196] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
I20240503 14:12:27.553925 30142 ts_tablet_manager.cc:1909] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.555624 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.555501 29828 log.cc:1196] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6d6b50aed01a4bf8af8889036dd83606
I20240503 14:12:27.556000 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 4fad4508a29d4822b81af6947b57e7ef (table default.Table_68a22cfee7f644fcba9b9c3e34e6205c [id=0a3978674b74406484d7b5e9880f6220]) successfully deleted
W20240503 14:12:27.556370 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5de9edfb0d064939bda2ec7a9e0e500b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9edfb0d064939bda2ec7a9e0e500b already in progress: creating tablet
W20240503 14:12:27.556581 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0d4c5648ddc84407a8d8b44f2d63dcee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d4c5648ddc84407a8d8b44f2d63dcee already in progress: creating tablet
I20240503 14:12:27.556667 29828 ts_tablet_manager.cc:1909] T 6d6b50aed01a4bf8af8889036dd83606 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:27.556790 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 19d43cc864064c4c907ec0629391da10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19d43cc864064c4c907ec0629391da10 already in progress: creating tablet
W20240503 14:12:27.557016 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 13f8553cfbab4af2a38a73e81fed2a5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13f8553cfbab4af2a38a73e81fed2a5f already in progress: creating tablet
W20240503 14:12:27.557226 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9f5b973ea16d42babd2cc75e32624854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f5b973ea16d42babd2cc75e32624854 already in progress: creating tablet
I20240503 14:12:27.555120 28600 tablet_bootstrap.cc:492] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.557564 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 967) took 1583 ms. Trace:
I20240503 14:12:27.557658 29828 rpcz_store.cc:276] 0503 14:12:25.974344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.974393 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:27.447453 (+1473060us) ts_tablet_manager.cc:1120] Deleting tablet 6d6b50aed01a4bf8af8889036dd83606
0503 14:12:27.447463 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.485054 (+ 37591us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.543768 (+ 58714us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.557540 (+ 13772us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472893}
I20240503 14:12:27.557803 28600 ts_tablet_manager.cc:1367] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240503 14:12:27.558030 29828 tablet_replica.cc:330] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:27.558249 29828 raft_consensus.cc:2229] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.558456 29828 raft_consensus.cc:2258] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.558727 28604 tablet_bootstrap.cc:654] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.558921 28600 raft_consensus.cc:348] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.559069 29828 ts_tablet_manager.cc:1875] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.559196 28600 raft_consensus.cc:374] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.559307 28600 raft_consensus.cc:729] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.559605 28600 consensus_queue.cc:260] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.560221 28600 ts_tablet_manager.cc:1398] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:27.560463 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet b97da778da214bb6bc147c5ada6c7603 with delete_type TABLET_DATA_DELETED (replaced by tablet cc815484ff24435fab5d994e4e564668) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.560564 28600 tablet_bootstrap.cc:492] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.560978 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.561689 28326 tablet_service.cc:1467] Processing CreateTablet for tablet e23195428df94abf94100ac75814449f (DEFAULT_TABLE table=default.Table_41235c2e757a4d39925d2d8e6dc69514 [id=184e48182d0047019679fe666cd4cb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.561892 28600 tablet_bootstrap.cc:654] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.562054 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23195428df94abf94100ac75814449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.568485 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 966) took 1361 ms. Trace:
I20240503 14:12:27.568804 30142 rpcz_store.cc:276] 0503 14:12:26.206919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.206979 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:27.538667 (+1331688us) ts_tablet_manager.cc:1120] Deleting tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
0503 14:12:27.538682 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.543856 (+ 5174us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.553109 (+ 9253us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.568453 (+ 15344us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36,"tablet-delete.queue_time_us":1331456}
I20240503 14:12:27.569275 30142 tablet_replica.cc:330] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.569522 30142 raft_consensus.cc:2229] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.569801 30142 pending_rounds.cc:62] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:27.569903 30142 pending_rounds.cc:68] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597758284701696 op_type: NO_OP noop_request { }
I20240503 14:12:27.570068 30142 raft_consensus.cc:2866] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240503 14:12:27.547327 28150 leader_election.cc:341] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.570250 30142 raft_consensus.cc:2258] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.570345 28150 leader_election.cc:302] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.570655 30272 raft_consensus.cc:2726] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.570982 30142 ts_tablet_manager.cc:1875] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:27.571734 28150 leader_election.cc:341] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.578229 29994 consensus_queue.cc:1035] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:27.579674 28427 leader_election.cc:341] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:27.583984 28290 leader_election.cc:341] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 7b6c998e518541248a87e4c1551ec080
I20240503 14:12:27.584218 28290 leader_election.cc:302] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.584550 30263 raft_consensus.cc:2726] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:27.545540 28614 tablet_bootstrap.cc:654] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.545809 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4846d662a64d51b16fd5447c371e9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.583729 30272 raft_consensus.cc:482] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.586244 30272 raft_consensus.cc:504] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.587152 30142 ts_tablet_manager.cc:1888] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.591362 29828 ts_tablet_manager.cc:1888] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.592741 29994 raft_consensus.cc:482] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.592931 29994 raft_consensus.cc:504] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.587225 30272 leader_election.cc:288] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.593521 30142 log.cc:1196] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0a50b3e199314bfc8465fdb7b2425d07
I20240503 14:12:27.593633 29994 leader_election.cc:288] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.593912 30142 ts_tablet_manager.cc:1909] T 0a50b3e199314bfc8465fdb7b2425d07 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.595353 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 967) took 1387 ms. Trace:
I20240503 14:12:27.596241 29828 log.cc:1196] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/4fad4508a29d4822b81af6947b57e7ef
I20240503 14:12:27.596421 30142 rpcz_store.cc:276] 0503 14:12:26.207615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.207664 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:27.569216 (+1361552us) ts_tablet_manager.cc:1120] Deleting tablet 0a50b3e199314bfc8465fdb7b2425d07
0503 14:12:27.569229 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.581504 (+ 12275us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.587023 (+ 5519us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.595329 (+ 8306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361351}
I20240503 14:12:27.596819 30142 tablet_replica.cc:330] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.597020 30142 raft_consensus.cc:2229] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.597172 30142 raft_consensus.cc:2258] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.597740 30142 ts_tablet_manager.cc:1875] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.598112 29828 ts_tablet_manager.cc:1909] T 4fad4508a29d4822b81af6947b57e7ef P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:27.601181 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 968) took 1626 ms. Trace:
I20240503 14:12:27.603058 29828 rpcz_store.cc:276] 0503 14:12:25.974696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.975157 (+ 461us) service_pool.cc:225] Handling call
0503 14:12:27.557972 (+1582815us) ts_tablet_manager.cc:1120] Deleting tablet 4fad4508a29d4822b81af6947b57e7ef
0503 14:12:27.557982 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.580428 (+ 22446us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.591234 (+ 10806us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.601152 (+ 9918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582658}
I20240503 14:12:27.603722 28600 tablet_bootstrap.cc:492] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.603893 28600 ts_tablet_manager.cc:1367] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20240503 14:12:27.604152 29828 tablet_replica.cc:330] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:27.604269 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e14e8b5f4046029af4673a277112d9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.604502 28212 raft_consensus.cc:2454] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.604745 28604 tablet_bootstrap.cc:492] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.604648 28600 raft_consensus.cc:348] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.604826 28600 raft_consensus.cc:374] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.604882 28600 raft_consensus.cc:729] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.604907 28604 ts_tablet_manager.cc:1367] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.003s
I20240503 14:12:27.604956 28427 leader_election.cc:302] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.605100 28600 consensus_queue.cc:260] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.605315 29994 raft_consensus.cc:2781] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.605397 29994 raft_consensus.cc:482] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.605464 29994 raft_consensus.cc:3037] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.605631 28604 raft_consensus.cc:348] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.605760 28600 ts_tablet_manager.cc:1398] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:27.605839 28604 raft_consensus.cc:374] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.605901 28604 raft_consensus.cc:729] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.605260 29828 raft_consensus.cc:2229] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.606103 28600 tablet_bootstrap.cc:492] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.606102 28604 consensus_queue.cc:260] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.606657 29828 raft_consensus.cc:2258] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.607877 28600 tablet_bootstrap.cc:654] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.608887 28604 ts_tablet_manager.cc:1398] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:27.609170 28604 tablet_bootstrap.cc:492] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.608052 29994 raft_consensus.cc:504] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.610241 29994 leader_election.cc:288] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.610695 28604 tablet_bootstrap.cc:654] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.611119 29828 ts_tablet_manager.cc:1875] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.611752 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e14e8b5f4046029af4673a277112d9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.612010 28212 raft_consensus.cc:3037] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.612354 28661 ts_tablet_manager.cc:1888] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.614089 28661 log.cc:1196] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/b2bba6a3d6e84552b027f23a29cbb38a
I20240503 14:12:27.614516 28661 ts_tablet_manager.cc:1909] T b2bba6a3d6e84552b027f23a29cbb38a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.614791 28212 raft_consensus.cc:2454] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.615695 28427 leader_election.cc:302] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.615844 30142 ts_tablet_manager.cc:1888] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.616183 29994 raft_consensus.cc:2781] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.616390 29994 raft_consensus.cc:686] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:27.616706 29994 consensus_queue.cc:237] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:27.619570 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 62355.5us
I20240503 14:12:27.619967 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 11ac912a3bd64cdc9082596e09c72e13 (table default.Table_78f62674ace541fabf9047f21a7cadbc [id=aed2a71b61da4d3981c77404ba8dffaa]) successfully deleted
I20240503 14:12:27.620352 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a (table default.Table_11703c3db99a47cc85741ba32923f695 [id=7448b8a18a6d45c48bc56d1379672494]) successfully deleted
I20240503 14:12:27.620658 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0a50b3e199314bfc8465fdb7b2425d07 (table default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0 [id=82ad92c83a0c44bc86486fb6137a47e3]) successfully deleted
W20240503 14:12:27.587582 28150 leader_election.cc:341] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.621701 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad43f3c6094253ba87d0ed3fc29fd9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.622061 28469 tablet_service.cc:1467] Processing CreateTablet for tablet e23195428df94abf94100ac75814449f (DEFAULT_TABLE table=default.Table_41235c2e757a4d39925d2d8e6dc69514 [id=184e48182d0047019679fe666cd4cb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.622535 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23195428df94abf94100ac75814449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.623407 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 980) took 1885 ms. Trace:
I20240503 14:12:27.623493 28661 rpcz_store.cc:276] 0503 14:12:25.737891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.738030 (+ 139us) service_pool.cc:225] Handling call
0503 14:12:27.522094 (+1784064us) ts_tablet_manager.cc:1120] Deleting tablet b2bba6a3d6e84552b027f23a29cbb38a
0503 14:12:27.522107 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.594699 (+ 72592us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.612240 (+ 17541us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.623384 (+ 11144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6885,"tablet-delete.queue_time_us":1783949}
I20240503 14:12:27.623958 28661 tablet_replica.cc:330] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.624181 28661 raft_consensus.cc:2229] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.624369 28661 raft_consensus.cc:2258] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.624809 28196 tablet_service.cc:1467] Processing CreateTablet for tablet e23195428df94abf94100ac75814449f (DEFAULT_TABLE table=default.Table_41235c2e757a4d39925d2d8e6dc69514 [id=184e48182d0047019679fe666cd4cb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.625207 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23195428df94abf94100ac75814449f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.627351 28604 tablet_bootstrap.cc:492] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.627532 28604 ts_tablet_manager.cc:1367] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240503 14:12:27.628398 28604 raft_consensus.cc:348] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.628479 28466 tablet_service.cc:1467] Processing CreateTablet for tablet bf1cdad717bf465b83598786801da37f (DEFAULT_TABLE table=default.Table_5f5dadbbef444ab2b13677a8e8879d4d [id=10f6ada2407446f387b729bacedb8283]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.628607 28604 raft_consensus.cc:374] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.628655 28604 raft_consensus.cc:729] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.628923 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1cdad717bf465b83598786801da37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.629647 28600 tablet_bootstrap.cc:492] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.629951 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d45788ae644a8581acca54cc55dd3f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.630429 28604 consensus_queue.cc:260] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:27.630208 28150 leader_election.cc:341] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.631071 28604 ts_tablet_manager.cc:1398] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:27.631381 28604 tablet_bootstrap.cc:492] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:27.631382 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 47410us
I20240503 14:12:27.631778 28290 leader_election.cc:302] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.632113 28600 ts_tablet_manager.cc:1367] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20240503 14:12:27.632062 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad43f3c6094253ba87d0ed3fc29fd9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:27.632709 28150 leader_election.cc:341] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.632958 28600 raft_consensus.cc:348] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.633996 30263 raft_consensus.cc:2726] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 14:12:27.634696 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d45788ae644a8581acca54cc55dd3f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:27.634841 28600 raft_consensus.cc:374] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.634919 28600 raft_consensus.cc:729] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.635116 28600 consensus_queue.cc:260] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.637045 28604 tablet_bootstrap.cc:654] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.637722 28614 tablet_bootstrap.cc:492] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:27.637871 28614 ts_tablet_manager.cc:1367] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.002s
I20240503 14:12:27.638154 28600 ts_tablet_manager.cc:1398] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20240503 14:12:27.639099 28150 leader_election.cc:341] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.639482 29996 raft_consensus.cc:482] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.639564 30227 tablet.cc:1722] T 77c77a11ef384709aef5b62273380573 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:27.639613 29996 raft_consensus.cc:504] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.640275 29996 leader_election.cc:288] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.644454 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0175b38c1e0464a9a6208cfaf68ca0d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.640306 28600 tablet_bootstrap.cc:492] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.640389 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 544b59c536bf40878c989c1328a5e618 (DEFAULT_TABLE table=default.Table_2be14bca3b2b481c93119dbb4ffb6590 [id=6e556b2c2b3441c890a32f7cc568dc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.645337 30142 log.cc:1196] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/3d54cc5c03034a6685c1da34f9bd65ac
I20240503 14:12:27.645449 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544b59c536bf40878c989c1328a5e618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.639232 28150 leader_election.cc:302] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.646538 28600 tablet_bootstrap.cc:654] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.641010 28197 tablet_service.cc:1467] Processing CreateTablet for tablet b8e5cb51cd764219ab3498b74aee7db7 (DEFAULT_TABLE table=default.Table_28f6161ffee747d8a270da263c869b2b [id=693ccf3d65e24d4ab1b32ff9b07abadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.648545 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e5cb51cd764219ab3498b74aee7db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.649571 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 544b59c536bf40878c989c1328a5e618 (DEFAULT_TABLE table=default.Table_2be14bca3b2b481c93119dbb4ffb6590 [id=6e556b2c2b3441c890a32f7cc568dc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.649966 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544b59c536bf40878c989c1328a5e618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.650190 30142 ts_tablet_manager.cc:1909] T 3d54cc5c03034a6685c1da34f9bd65ac P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.650750 30272 raft_consensus.cc:2726] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:27.651310 28150 leader_election.cc:341] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.651443 28150 leader_election.cc:302] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.651693 30272 raft_consensus.cc:2726] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.641611 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 01482b478dcd45598fa65d14904fe276 (DEFAULT_TABLE table=default.Table_7fb65f72628b40a4855474fe07921278 [id=28d97a0a80dc4e0bbea52f5dd8e32080]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.652671 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01482b478dcd45598fa65d14904fe276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.654942 28456 tablet_service.cc:1467] Processing CreateTablet for tablet b8e5cb51cd764219ab3498b74aee7db7 (DEFAULT_TABLE table=default.Table_28f6161ffee747d8a270da263c869b2b [id=693ccf3d65e24d4ab1b32ff9b07abadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.655337 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e5cb51cd764219ab3498b74aee7db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.642187 28188 tablet_service.cc:1467] Processing CreateTablet for tablet beba0d09f5d34980ba7ed463ff9b4c34 (DEFAULT_TABLE table=default.Table_70505af08e3d4b4aa9b097f33cae154a [id=b575b5aeba3545c6bf6d5123c857b546]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.657728 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beba0d09f5d34980ba7ed463ff9b4c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.642803 28186 tablet_service.cc:1467] Processing CreateTablet for tablet f7d18fdc987d4879a642438e748eeb62 (DEFAULT_TABLE table=default.Table_4fb75e6e800e495c9dfda4864bdebe5a [id=0b0f60024a7b467b97ed1a22e404268d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.660560 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d18fdc987d4879a642438e748eeb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.662967 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 968) took 1454 ms. Trace:
I20240503 14:12:27.663067 30142 rpcz_store.cc:276] 0503 14:12:26.208218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.208267 (+ 49us) service_pool.cc:225] Handling call
0503 14:12:27.596761 (+1388494us) ts_tablet_manager.cc:1120] Deleting tablet 3d54cc5c03034a6685c1da34f9bd65ac
0503 14:12:27.596772 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.612389 (+ 15617us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.615723 (+ 3334us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.662941 (+ 47218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11820,"tablet-delete.queue_time_us":1388280}
I20240503 14:12:27.663477 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 969) took 1453 ms. Trace:
I20240503 14:12:27.663554 30142 rpcz_store.cc:276] 0503 14:12:26.209793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.209878 (+ 85us) service_pool.cc:225] Handling call
0503 14:12:27.663353 (+1453475us) ts_tablet_manager.cc:1120] Deleting tablet 8c6202333af943cbbc0d09553c1c193c
0503 14:12:27.663365 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.663454 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1453276}
I20240503 14:12:27.663849 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 977) took 1274 ms. Trace:
I20240503 14:12:27.663906 30142 rpcz_store.cc:276] 0503 14:12:26.389626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.389688 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:27.663750 (+1274062us) ts_tablet_manager.cc:1120] Deleting tablet addc2a6195034b0a927fdee7a2955122
0503 14:12:27.663758 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.663836 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1273585}
I20240503 14:12:27.664227 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 978) took 1183 ms. Trace:
I20240503 14:12:27.639621 28180 tablet_service.cc:1467] Processing CreateTablet for tablet bf1cdad717bf465b83598786801da37f (DEFAULT_TABLE table=default.Table_5f5dadbbef444ab2b13677a8e8879d4d [id=10f6ada2407446f387b729bacedb8283]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.664306 30142 rpcz_store.cc:276] 0503 14:12:26.481032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.481087 (+ 55us) service_pool.cc:225] Handling call
0503 14:12:27.664071 (+1182984us) ts_tablet_manager.cc:1120] Deleting tablet 77511682d71a40f5933edcb6a6523649
0503 14:12:27.664078 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.664208 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1182736}
I20240503 14:12:27.664587 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1cdad717bf465b83598786801da37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.664629 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 979) took 1113 ms. Trace:
I20240503 14:12:27.664688 30142 rpcz_store.cc:276] 0503 14:12:26.551451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.551668 (+ 217us) service_pool.cc:225] Handling call
0503 14:12:27.664530 (+1112862us) ts_tablet_manager.cc:1120] Deleting tablet c99c15b4d13647e98cae1375b7f13827
0503 14:12:27.664538 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.664615 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110492}
I20240503 14:12:27.664986 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 980) took 1111 ms. Trace:
I20240503 14:12:27.665050 30142 rpcz_store.cc:276] 0503 14:12:26.553627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.553693 (+ 66us) service_pool.cc:225] Handling call
0503 14:12:27.664864 (+1111171us) ts_tablet_manager.cc:1120] Deleting tablet d2eb8328e86a4ee5a53647a810729519
0503 14:12:27.664871 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.664972 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110588}
I20240503 14:12:27.665341 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 981) took 1066 ms. Trace:
I20240503 14:12:27.665409 30142 rpcz_store.cc:276] 0503 14:12:26.598836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.598908 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:27.665222 (+1066314us) ts_tablet_manager.cc:1120] Deleting tablet 7b7ca19dd15f48a68ace7fefc7ae660f
0503 14:12:27.665230 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.665326 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1066073}
I20240503 14:12:27.665622 30142 tablet_replica.cc:330] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.665791 30142 raft_consensus.cc:2229] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.665969 30142 raft_consensus.cc:2258] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.666548 30142 ts_tablet_manager.cc:1875] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.668983 28604 tablet_bootstrap.cc:492] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.669134 28604 ts_tablet_manager.cc:1367] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20240503 14:12:27.669927 28604 raft_consensus.cc:348] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.670594 29996 raft_consensus.cc:482] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.670802 29996 raft_consensus.cc:504] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.671429 29996 leader_election.cc:288] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.671557 28604 raft_consensus.cc:374] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.671615 28604 raft_consensus.cc:729] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.671799 28604 consensus_queue.cc:260] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.672667 28604 ts_tablet_manager.cc:1398] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:27.672952 28604 tablet_bootstrap.cc:492] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.673633 28466 tablet_service.cc:1467] Processing CreateTablet for tablet beba0d09f5d34980ba7ed463ff9b4c34 (DEFAULT_TABLE table=default.Table_70505af08e3d4b4aa9b097f33cae154a [id=b575b5aeba3545c6bf6d5123c857b546]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.674041 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beba0d09f5d34980ba7ed463ff9b4c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.674645 28604 tablet_bootstrap.cc:654] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.676074 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 01482b478dcd45598fa65d14904fe276 (DEFAULT_TABLE table=default.Table_7fb65f72628b40a4855474fe07921278 [id=28d97a0a80dc4e0bbea52f5dd8e32080]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.677203 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01482b478dcd45598fa65d14904fe276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.677146 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcc6cef7b7d464a9fbbae61caf0437d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.677388 28215 raft_consensus.cc:2454] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.676462 28462 tablet_service.cc:1467] Processing CreateTablet for tablet f7d18fdc987d4879a642438e748eeb62 (DEFAULT_TABLE table=default.Table_4fb75e6e800e495c9dfda4864bdebe5a [id=0b0f60024a7b467b97ed1a22e404268d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.678632 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d18fdc987d4879a642438e748eeb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.679260 30227 tablet.cc:1722] T e94673f00cb3432185009bda2ce398fa P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:27.679931 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d1314841574fc8bcf3ccb16755e421" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.680848 28213 raft_consensus.cc:2454] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.689266 30318 tablet.cc:1722] T e94673f00cb3432185009bda2ce398fa P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:27.692054 30266 raft_consensus.cc:482] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.692235 30266 raft_consensus.cc:504] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.692837 30266 leader_election.cc:288] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.696698 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b2865e9604ce58280321c7ee2a88e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:27.697839 28150 leader_election.cc:341] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.698192 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b2865e9604ce58280321c7ee2a88e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:27.700645 28427 consensus_peers.cc:485] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20b176a071ba4f3fbff2a64d6e46027a. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:27.700596 28150 leader_election.cc:341] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.700767 28150 leader_election.cc:302] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.700852 28427 leader_election.cc:302] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.701025 30266 raft_consensus.cc:2726] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.701289 28604 tablet_bootstrap.cc:492] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.701449 28604 ts_tablet_manager.cc:1367] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20240503 14:12:27.701647 29996 raft_consensus.cc:2781] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.701738 29996 raft_consensus.cc:482] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.701814 29996 raft_consensus.cc:3037] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.702224 28604 raft_consensus.cc:348] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.702432 28604 raft_consensus.cc:374] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.702492 28604 raft_consensus.cc:729] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.702698 28604 consensus_queue.cc:260] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.703707 28600 tablet_bootstrap.cc:492] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.703866 28600 ts_tablet_manager.cc:1367] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.006s
I20240503 14:12:27.704438 29996 raft_consensus.cc:504] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.704648 28600 raft_consensus.cc:348] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.704855 28600 raft_consensus.cc:374] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.704913 28600 raft_consensus.cc:729] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.704994 28661 ts_tablet_manager.cc:1875] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.705103 28600 consensus_queue.cc:260] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.705107 29996 leader_election.cc:288] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.708029 28600 ts_tablet_manager.cc:1398] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:27.708388 28600 tablet_bootstrap.cc:492] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.709443 30319 tablet.cc:1722] T e94673f00cb3432185009bda2ce398fa P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:27.710068 28600 tablet_bootstrap.cc:654] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.714527 28604 ts_tablet_manager.cc:1398] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:12:27.714885 28604 tablet_bootstrap.cc:492] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.716029 28427 leader_election.cc:302] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.716351 30253 raft_consensus.cc:2781] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.716440 30253 raft_consensus.cc:482] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.716509 30253 raft_consensus.cc:3037] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.718298 30142 ts_tablet_manager.cc:1888] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:27.718672 30142 log.cc:1196] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/5b90e9563b4c4cef8ee7964eaa96e285
I20240503 14:12:27.720257 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcc6cef7b7d464a9fbbae61caf0437d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.720461 28213 raft_consensus.cc:3037] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.723084 28213 raft_consensus.cc:2454] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.719036 30253 raft_consensus.cc:504] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.723805 30142 ts_tablet_manager.cc:1909] T 5b90e9563b4c4cef8ee7964eaa96e285 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.724824 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 982) took 1098 ms. Trace:
I20240503 14:12:27.725492 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e14e8b5f4046029af4673a277112d9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.726845 30142 rpcz_store.cc:276] 0503 14:12:26.626057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.626122 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:27.665584 (+1039462us) ts_tablet_manager.cc:1120] Deleting tablet 5b90e9563b4c4cef8ee7964eaa96e285
0503 14:12:27.665591 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.704386 (+ 38795us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.718141 (+ 13755us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.724794 (+ 6653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039256}
I20240503 14:12:27.727511 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6d6b50aed01a4bf8af8889036dd83606 (table default.Table_ede6e833710649e08e31516f0890c2d6 [id=3d49172b73ba44538840b651a85b24e3]) successfully deleted
I20240503 14:12:27.728293 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 983) took 1101 ms. Trace:
I20240503 14:12:27.728394 30142 rpcz_store.cc:276] 0503 14:12:26.626943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.626993 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:27.727203 (+1100210us) ts_tablet_manager.cc:1120] Deleting tablet 7d348da459634561985c4ac1710c0bb2
0503 14:12:27.728124 (+ 921us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.728271 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100027}
I20240503 14:12:27.729617 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e14e8b5f4046029af4673a277112d9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.730105 28614 raft_consensus.cc:348] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.730381 28614 raft_consensus.cc:374] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.730473 28614 raft_consensus.cc:729] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.730782 28614 consensus_queue.cc:260] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.731341 30266 raft_consensus.cc:482] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.731415 28614 ts_tablet_manager.cc:1398] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.093s user 0.001s sys 0.001s
I20240503 14:12:27.731500 30266 raft_consensus.cc:504] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.731714 28614 tablet_bootstrap.cc:492] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:27.732184 30266 leader_election.cc:288] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.732723 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816f3b05cfb4496984f6cb28d310cc4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.732815 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816f3b05cfb4496984f6cb28d310cc4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:27.733084 28488 raft_consensus.cc:2454] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
W20240503 14:12:27.733350 28150 leader_election.cc:341] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.733594 28150 leader_election.cc:302] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:27.733824 30253 leader_election.cc:288] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.733853 30266 raft_consensus.cc:2781] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.733927 30266 raft_consensus.cc:482] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.733994 30266 raft_consensus.cc:3037] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.734233 30142 tablet_replica.cc:330] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.734243 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcc6cef7b7d464a9fbbae61caf0437d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.734417 30142 raft_consensus.cc:2229] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.734434 28352 raft_consensus.cc:2454] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.734552 30142 raft_consensus.cc:2258] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.734793 28604 tablet_bootstrap.cc:654] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.734861 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d1314841574fc8bcf3ccb16755e421" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.735069 30142 ts_tablet_manager.cc:1875] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.735348 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcc6cef7b7d464a9fbbae61caf0437d" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.735491 28352 raft_consensus.cc:3037] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.736869 30266 raft_consensus.cc:504] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.737622 30266 leader_election.cc:288] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.738124 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816f3b05cfb4496984f6cb28d310cc4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:27.738332 30266 raft_consensus.cc:482] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.738328 28352 raft_consensus.cc:2454] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:27.738449 30266 raft_consensus.cc:504] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.739068 30266 leader_election.cc:288] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:27.739624 28150 leader_election.cc:341] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.739879 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:27.740415 28150 leader_election.cc:341] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.740824 28600 tablet_bootstrap.cc:492] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.741005 28600 ts_tablet_manager.cc:1367] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240503 14:12:27.740983 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:27.741806 28150 leader_election.cc:341] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.741916 28150 leader_election.cc:302] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.742043 28600 raft_consensus.cc:348] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.742239 28600 raft_consensus.cc:374] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.742298 28600 raft_consensus.cc:729] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.742622 30266 raft_consensus.cc:2726] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.743870 30266 raft_consensus.cc:482] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.743989 30266 raft_consensus.cc:504] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.744601 30266 leader_election.cc:288] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.742527 28600 consensus_queue.cc:260] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.745121 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e87faea1415d95ae014a93fa22af" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:27.745137 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e87faea1415d95ae014a93fa22af" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:27.746001 28150 leader_election.cc:341] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.746053 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816f3b05cfb4496984f6cb28d310cc4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
W20240503 14:12:27.746177 28150 leader_election.cc:341] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.746237 28488 raft_consensus.cc:3037] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.746245 28150 leader_election.cc:302] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.746707 28600 ts_tablet_manager.cc:1398] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:27.747051 28600 tablet_bootstrap.cc:492] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.748775 28488 raft_consensus.cc:2454] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:27.749140 28150 leader_election.cc:302] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:27.749552 30272 raft_consensus.cc:2726] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.749722 30272 raft_consensus.cc:2781] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.749820 30272 raft_consensus.cc:686] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:27.750061 30272 consensus_queue.cc:237] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.750754 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7549b2ca31b04247bf3a2202e7585817
I20240503 14:12:27.751581 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_928c3ae55b104d519b4ad071f8babaac" } new_table_name: "default.Table_3d6e2487693e459795af13f1a020a576" modify_external_catalogs: true
I20240503 14:12:27.752035 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 3dfd6d40a57e4bea973874ff12c0ea3a
I20240503 14:12:27.752794 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8ce861018efe45e58f9c7ed20c85a6d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.753374 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ce861018efe45e58f9c7ed20c85a6d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.757390 30142 ts_tablet_manager.cc:1888] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.757524 30142 log.cc:1196] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/dbcc6cef7b7d464a9fbbae61caf0437d
I20240503 14:12:27.757844 30142 ts_tablet_manager.cc:1909] T dbcc6cef7b7d464a9fbbae61caf0437d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.758049 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5500da0e5658492b840dd29273ab265f
I20240503 14:12:27.758459 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 5500da0e5658492b840dd29273ab265f deleted and replaced by tablet a0f4738ccaad47ab87317ee90efcd46c
I20240503 14:12:27.758692 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet e220336186f648419b186feada6f0124 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:27.758889 30142 tablet_replica.cc:330] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.759265 30142 raft_consensus.cc:2229] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.759431 30142 raft_consensus.cc:2258] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.759951 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 5685a6b3ea6943f8a72d74de87ac2166
I20240503 14:12:27.759999 30142 ts_tablet_manager.cc:1875] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.760694 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_84bf4809af6d4c26bdb71f0ae680cfb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.761281 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84bf4809af6d4c26bdb71f0ae680cfb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.762493 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 91eb4433685b4119a170054a0a777e05
I20240503 14:12:27.763211 29828 ts_tablet_manager.cc:1888] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.763324 29828 log.cc:1196] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/8fe3affd0d144b1fb0078e7690b9b36d
I20240503 14:12:27.763182 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b3d87c07d8a8429693540260e15e7362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:27.763697 29828 ts_tablet_manager.cc:1909] T 8fe3affd0d144b1fb0078e7690b9b36d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:27.763744 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3d87c07d8a8429693540260e15e7362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.764603 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 969) took 1789 ms. Trace:
I20240503 14:12:27.764706 29828 rpcz_store.cc:276] 0503 14:12:25.975454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.975489 (+ 35us) service_pool.cc:225] Handling call
0503 14:12:27.604055 (+1628566us) ts_tablet_manager.cc:1120] Deleting tablet 8fe3affd0d144b1fb0078e7690b9b36d
0503 14:12:27.604067 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.718338 (+114271us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.763086 (+ 44748us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.764556 (+ 1470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628419}
I20240503 14:12:27.765058 29828 tablet_replica.cc:330] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:27.765273 29828 raft_consensus.cc:2229] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.765249 28025 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6b099d92a07649ee821a8507248767ed" } new_table_name: "default.Table_392e46b937cf45e58d01188a21cf276c" modify_external_catalogs: true
I20240503 14:12:27.765460 29828 raft_consensus.cc:2258] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.766005 30142 ts_tablet_manager.cc:1888] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.766119 30142 log.cc:1196] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a884cf4ea11e48ef941722b79ad50075
I20240503 14:12:27.766434 30142 ts_tablet_manager.cc:1909] T a884cf4ea11e48ef941722b79ad50075 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.767232 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 988) took 1001 ms. Trace:
I20240503 14:12:27.767323 30142 rpcz_store.cc:276] 0503 14:12:26.765953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.766830 (+ 877us) service_pool.cc:225] Handling call
0503 14:12:27.758833 (+992003us) ts_tablet_manager.cc:1120] Deleting tablet a884cf4ea11e48ef941722b79ad50075
0503 14:12:27.758843 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.762844 (+ 4001us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.765892 (+ 3048us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.767203 (+ 1311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":986842}
I20240503 14:12:27.767866 30142 tablet_replica.cc:330] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.768062 30142 raft_consensus.cc:2229] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.768302 30142 raft_consensus.cc:2258] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.768842 30142 ts_tablet_manager.cc:1875] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.769780 28661 ts_tablet_manager.cc:1888] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.769904 28661 log.cc:1196] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/3d137ef4cfc2476cbc70f1cc72ad89e6
I20240503 14:12:27.770268 28661 ts_tablet_manager.cc:1909] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.770972 28029 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 3e7b2865e9604ce58280321c7ee2a88e (table default.Table_ca6cee392d934dd3ac773e100ae031d2 [id=32be054076954aed9265888ca62e510b]). Expected version 4 got 0
I20240503 14:12:27.771078 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 982) took 2032 ms. Trace:
I20240503 14:12:27.771159 28661 rpcz_store.cc:276] 0503 14:12:25.738103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.738142 (+ 39us) service_pool.cc:225] Handling call
0503 14:12:27.623903 (+1885761us) ts_tablet_manager.cc:1120] Deleting tablet 3d137ef4cfc2476cbc70f1cc72ad89e6
0503 14:12:27.623915 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.720740 (+ 96825us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.769648 (+ 48908us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.771050 (+ 1402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885603}
I20240503 14:12:27.771503 28661 tablet_replica.cc:330] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.771704 28661 raft_consensus.cc:2229] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.771867 28661 raft_consensus.cc:2258] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.773285 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b554e9ad820f41199f4ee1e53469d221"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:27.773871 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b554e9ad820f41199f4ee1e53469d221 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.775246 30142 ts_tablet_manager.cc:1888] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.775362 30142 log.cc:1196] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/c9325d5142af4b018354bf38370b61e0
I20240503 14:12:27.775698 30142 ts_tablet_manager.cc:1909] T c9325d5142af4b018354bf38370b61e0 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.776342 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ae0e06cb36cd42238ccf6efe518640c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:27.776716 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 989) took 1010 ms. Trace:
I20240503 14:12:27.776804 30142 rpcz_store.cc:276] 0503 14:12:26.766032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.766914 (+ 882us) service_pool.cc:225] Handling call
0503 14:12:27.767810 (+1000896us) ts_tablet_manager.cc:1120] Deleting tablet c9325d5142af4b018354bf38370b61e0
0503 14:12:27.767821 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.771925 (+ 4104us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.775121 (+ 3196us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.776686 (+ 1565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":995777}
W20240503 14:12:27.776993 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae0e06cb36cd42238ccf6efe518640c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:27.777349 30142 tablet_replica.cc:330] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.777552 30142 raft_consensus.cc:2229] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.777707 30142 raft_consensus.cc:2258] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.778268 30142 ts_tablet_manager.cc:1875] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.784732 30142 ts_tablet_manager.cc:1888] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.785236 30142 log.cc:1196] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e0175b38c1e0464a9a6208cfaf68ca0d
I20240503 14:12:27.785665 30142 ts_tablet_manager.cc:1909] T e0175b38c1e0464a9a6208cfaf68ca0d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.786512 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 990) took 1020 ms. Trace:
I20240503 14:12:27.786665 30142 rpcz_store.cc:276] 0503 14:12:26.766107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.767003 (+ 896us) service_pool.cc:225] Handling call
0503 14:12:27.777292 (+1010289us) ts_tablet_manager.cc:1120] Deleting tablet e0175b38c1e0464a9a6208cfaf68ca0d
0503 14:12:27.777304 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.781486 (+ 4182us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.784623 (+ 3137us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.786487 (+ 1864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005221}
I20240503 14:12:27.787372 30142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 992) took 1006 ms. Trace:
I20240503 14:12:27.787624 30142 rpcz_store.cc:276] 0503 14:12:26.781323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.781771 (+ 448us) service_pool.cc:225] Handling call
0503 14:12:27.787202 (+1005431us) ts_tablet_manager.cc:1120] Deleting tablet bab214b7ec58425d8cd1180ee16b273d
0503 14:12:27.787214 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.787351 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005213}
I20240503 14:12:27.766077 29828 ts_tablet_manager.cc:1875] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.788646 30142 tablet_replica.cc:330] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.788893 30142 raft_consensus.cc:2229] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.789084 30142 raft_consensus.cc:2258] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.789680 30142 ts_tablet_manager.cc:1875] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:27.792714 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4cda92c8e3c94139b84eabc81127613e consensus state has no leader
I20240503 14:12:27.796448 30142 ts_tablet_manager.cc:1888] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.796694 28661 ts_tablet_manager.cc:1875] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.796734 30142 log.cc:1196] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ce4846d662a64d51b16fd5447c371e9a
I20240503 14:12:27.797211 30142 ts_tablet_manager.cc:1909] T ce4846d662a64d51b16fd5447c371e9a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.800930 30142 tablet_replica.cc:330] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.801115 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 86d810ab0cf34941a343eef1593632bf (table default.Table_9f2d3cc9c6724dab9ba35ff24e85c236 [id=c358e28088d44cf598c96a1bd2a4321e]). Expected version 1 got 0
I20240503 14:12:27.801260 30142 raft_consensus.cc:2229] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.801493 30142 raft_consensus.cc:2258] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.801510 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:27.801712 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 77c77a11ef384709aef5b62273380573 (table default.Table_ece674e6e74547b6a4d04f1f2b167a08 [id=2edd1164c07a4a67b51262d402d53596]). Expected version 3 got 0
I20240503 14:12:27.802125 30142 ts_tablet_manager.cc:1875] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.803802 28600 tablet_bootstrap.cc:654] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.807880 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d1314841574fc8bcf3ccb16755e421" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:27.808914 30142 ts_tablet_manager.cc:1888] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.809113 30142 log.cc:1196] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a3d1314841574fc8bcf3ccb16755e421
I20240503 14:12:27.809535 30142 ts_tablet_manager.cc:1909] T a3d1314841574fc8bcf3ccb16755e421 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.810765 30142 tablet_replica.cc:330] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.811051 30142 raft_consensus.cc:2229] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.811303 30142 raft_consensus.cc:2258] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.811955 30142 ts_tablet_manager.cc:1875] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:27.813009 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 65694.1us
I20240503 14:12:27.814863 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 9cad0a4cdfa8471bbd7066b91e85022e (DEFAULT_TABLE table=default.Table_b15ac98442fe4481b0495a342ae6a57a [id=06b7f1e982194749a1caf59ad3464e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.815315 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cad0a4cdfa8471bbd7066b91e85022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.817126 28600 tablet_bootstrap.cc:492] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.817278 28600 ts_tablet_manager.cc:1367] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.003s
I20240503 14:12:27.818194 28600 raft_consensus.cc:348] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.818404 28600 raft_consensus.cc:374] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.818459 28600 raft_consensus.cc:729] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.819200 30142 ts_tablet_manager.cc:1888] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.818698 28600 consensus_queue.cc:260] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.819471 30142 log.cc:1196] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:27.820787 30142 ts_tablet_manager.cc:1909] T 62e4f6e3908e420e9fa156ee24b437d6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.821157 29828 ts_tablet_manager.cc:1888] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.822279 28600 ts_tablet_manager.cc:1398] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:12:27.822522 28427 leader_election.cc:302] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:27.822629 28600 tablet_bootstrap.cc:492] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.822856 30253 raft_consensus.cc:2781] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.822980 30253 raft_consensus.cc:686] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:27.823230 30253 consensus_queue.cc:237] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.824352 28600 tablet_bootstrap.cc:654] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.825649 30142 tablet_replica.cc:330] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.825896 30142 raft_consensus.cc:2229] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.826083 30142 raft_consensus.cc:2258] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.826815 30142 ts_tablet_manager.cc:1875] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.836458 30142 ts_tablet_manager.cc:1888] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.836654 30142 log.cc:1196] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/13c7a51c68ef478e9bdc5b3e3dc3d5df
I20240503 14:12:27.837023 30142 ts_tablet_manager.cc:1909] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.837666 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbc326f0a9649ce9fe5976a8fb895f5 (DEFAULT_TABLE table=default.Table_d94a92f094354a9abb3e0f0173679568 [id=f187e80171a14052b5b91e3372b75d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.838219 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbc326f0a9649ce9fe5976a8fb895f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.839044 30142 tablet_replica.cc:330] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.839298 30142 raft_consensus.cc:2229] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.839521 29828 log.cc:1196] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ff4d28d3c2ea472191ad21355248a2fc
I20240503 14:12:27.839614 30142 pending_rounds.cc:62] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:27.839704 30142 pending_rounds.cc:68] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597762718248960 op_type: NO_OP noop_request { }
I20240503 14:12:27.839877 30142 raft_consensus.cc:2866] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:27.840020 30142 raft_consensus.cc:2258] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.840557 29828 ts_tablet_manager.cc:1909] T ff4d28d3c2ea472191ad21355248a2fc P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:27.841029 30142 ts_tablet_manager.cc:1875] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.844990 28600 tablet_bootstrap.cc:492] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.845199 28600 ts_tablet_manager.cc:1367] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 14:12:27.836433 30319 tablet.cc:1722] T 77c77a11ef384709aef5b62273380573 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:27.846077 28600 raft_consensus.cc:348] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.846310 28600 raft_consensus.cc:374] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.846385 28600 raft_consensus.cc:729] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.846633 28600 consensus_queue.cc:260] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.848336 30142 ts_tablet_manager.cc:1888] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.848539 30142 log.cc:1196] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/920b41e3b21347b393497de4d4e87469
I20240503 14:12:27.848529 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 970) took 1872 ms. Trace:
I20240503 14:12:27.848966 30142 ts_tablet_manager.cc:1909] T 920b41e3b21347b393497de4d4e87469 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.849524 28600 ts_tablet_manager.cc:1398] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:12:27.850358 28600 tablet_bootstrap.cc:492] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.839445 28661 ts_tablet_manager.cc:1888] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.850782 28661 log.cc:1196] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/dcd405fd84d44076b1792b7fcbe00e84
I20240503 14:12:27.851325 28661 ts_tablet_manager.cc:1909] T dcd405fd84d44076b1792b7fcbe00e84 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.848644 29828 rpcz_store.cc:276] 0503 14:12:25.975812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.976872 (+ 1060us) service_pool.cc:225] Handling call
0503 14:12:27.764985 (+1788113us) ts_tablet_manager.cc:1120] Deleting tablet ff4d28d3c2ea472191ad21355248a2fc
0503 14:12:27.764996 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.791476 (+ 26480us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.821023 (+ 29547us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.848480 (+ 27457us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787940}
I20240503 14:12:27.852030 28600 tablet_bootstrap.cc:654] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.852703 29828 tablet_replica.cc:330] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:27.854470 28604 tablet_bootstrap.cc:492] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.854681 28604 ts_tablet_manager.cc:1367] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.005s
W20240503 14:12:27.833402 28427 consensus_peers.cc:485] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57ab077988364bd687e54f90c3c265d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.855278 29994 consensus_queue.cc:1035] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.855641 28604 raft_consensus.cc:348] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.855808 28604 raft_consensus.cc:374] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.855867 28604 raft_consensus.cc:729] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.856047 28604 consensus_queue.cc:260] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.856712 28604 ts_tablet_manager.cc:1398] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:27.857009 28604 tablet_bootstrap.cc:492] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.859418 28488 raft_consensus.cc:3037] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.865064 28187 tablet_service.cc:1467] Processing CreateTablet for tablet adfd840fbfe7428bb9409964c120a906 (DEFAULT_TABLE table=default.Table_ddb29edceb57402988fa9a6aa01c4c1c [id=240a46975acc41fa8e8ad805b5c6f675]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:27.865826 28427 leader_election.cc:341] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:27.866195 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfd840fbfe7428bb9409964c120a906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.869591 29828 raft_consensus.cc:2229] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.870157 29828 raft_consensus.cc:2258] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.871044 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 983) took 2132 ms. Trace:
I20240503 14:12:27.871227 28661 rpcz_store.cc:276] 0503 14:12:25.738189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.738261 (+ 72us) service_pool.cc:225] Handling call
0503 14:12:27.771449 (+2033188us) ts_tablet_manager.cc:1120] Deleting tablet dcd405fd84d44076b1792b7fcbe00e84
0503 14:12:27.771462 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.799723 (+ 28261us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.839320 (+ 39597us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.871017 (+ 31697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2033056}
I20240503 14:12:27.878363 28186 tablet_service.cc:1467] Processing CreateTablet for tablet ae8b8922e42449ecb0b53805a8feba5b (DEFAULT_TABLE table=default.Table_3d13d5e860f2457895d61f215a2c08cf [id=9dc6978e6751487caf5a24a481876a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.878930 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8b8922e42449ecb0b53805a8feba5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.879395 28604 tablet_bootstrap.cc:654] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.858320 28614 tablet_bootstrap.cc:654] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.881623 29828 ts_tablet_manager.cc:1875] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.886914 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 7549b2ca31b04247bf3a2202e7585817 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.894708 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.895299 30315 raft_consensus.cc:482] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.895476 30315 raft_consensus.cc:504] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.896250 30315 leader_election.cc:288] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:27.896816 28661 tablet_replica.cc:330] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:27.897110 28661 raft_consensus.cc:2229] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:27.897542 28661 raft_consensus.cc:2258] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.898547 28661 ts_tablet_manager.cc:1875] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.899734 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 5500da0e5658492b840dd29273ab265f with delete_type TABLET_DATA_DELETED (replaced by tablet a0f4738ccaad47ab87317ee90efcd46c) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.902293 28326 tablet_service.cc:1467] Processing CreateTablet for tablet bf1cdad717bf465b83598786801da37f (DEFAULT_TABLE table=default.Table_5f5dadbbef444ab2b13677a8e8879d4d [id=10f6ada2407446f387b729bacedb8283]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.902787 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1cdad717bf465b83598786801da37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.905498 28661 ts_tablet_manager.cc:1888] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.905611 28661 log.cc:1196] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/07354861ee6142d6b1fbe573397d3160
I20240503 14:12:27.905920 28661 ts_tablet_manager.cc:1909] T 07354861ee6142d6b1fbe573397d3160 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:27.906698 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 981) took 2168 ms. Trace:
I20240503 14:12:27.906793 28661 rpcz_store.cc:276] 0503 14:12:25.738020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.738158 (+ 138us) service_pool.cc:225] Handling call
0503 14:12:27.871614 (+2133456us) ts_tablet_manager.cc:1120] Deleting tablet 07354861ee6142d6b1fbe573397d3160
0503 14:12:27.896179 (+ 24565us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.902189 (+ 6010us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.905378 (+ 3189us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.906673 (+ 1295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2133140}
I20240503 14:12:27.907243 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 984) took 2106 ms. Trace:
I20240503 14:12:27.907315 28661 rpcz_store.cc:276] 0503 14:12:25.800700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.800797 (+ 97us) service_pool.cc:225] Handling call
0503 14:12:27.907098 (+2106301us) ts_tablet_manager.cc:1120] Deleting tablet 6a27d21de6a6441d810f379bba88f9a2
0503 14:12:27.907108 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.907223 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2106123}
I20240503 14:12:27.907512 28604 tablet_bootstrap.cc:492] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.907655 28604 ts_tablet_manager.cc:1367] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240503 14:12:27.908537 28604 raft_consensus.cc:348] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.908761 28604 raft_consensus.cc:374] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.908818 28604 raft_consensus.cc:729] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.909041 28604 consensus_queue.cc:260] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.909374 29996 raft_consensus.cc:482] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.909492 29996 raft_consensus.cc:504] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.910091 29996 leader_election.cc:288] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:27.910534 28604 ts_tablet_manager.cc:1398] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:27.910832 28604 tablet_bootstrap.cc:492] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.911252 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 544b59c536bf40878c989c1328a5e618 (DEFAULT_TABLE table=default.Table_2be14bca3b2b481c93119dbb4ffb6590 [id=6e556b2c2b3441c890a32f7cc568dc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.911916 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544b59c536bf40878c989c1328a5e618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.912421 28604 tablet_bootstrap.cc:654] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.912701 30272 raft_consensus.cc:482] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.916222 28614 tablet_bootstrap.cc:492] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:27.916529 28614 ts_tablet_manager.cc:1367] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.185s user 0.004s sys 0.000s
I20240503 14:12:27.917445 28614 raft_consensus.cc:348] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.917734 28604 tablet_bootstrap.cc:492] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:27.917769 28614 raft_consensus.cc:374] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.917887 28604 ts_tablet_manager.cc:1367] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:27.918002 28614 raft_consensus.cc:729] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.918334 28614 consensus_queue.cc:260] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.918599 28604 raft_consensus.cc:348] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.918807 28604 raft_consensus.cc:374] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.918862 28604 raft_consensus.cc:729] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.919039 28604 consensus_queue.cc:260] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.922255 28614 ts_tablet_manager.cc:1398] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:27.922677 28614 tablet_bootstrap.cc:492] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:27.924317 28614 tablet_bootstrap.cc:654] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.925603 30266 raft_consensus.cc:482] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.925762 30266 raft_consensus.cc:504] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.926453 30266 leader_election.cc:288] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:27.927074 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:27.927757 28150 leader_election.cc:341] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.927093 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:27.930245 28150 leader_election.cc:341] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.930367 28150 leader_election.cc:302] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:27.930786 30266 raft_consensus.cc:2726] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.935751 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 5685a6b3ea6943f8a72d74de87ac2166 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.936652 28320 tablet_service.cc:1467] Processing CreateTablet for tablet b8e5cb51cd764219ab3498b74aee7db7 (DEFAULT_TABLE table=default.Table_28f6161ffee747d8a270da263c869b2b [id=693ccf3d65e24d4ab1b32ff9b07abadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.937170 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e5cb51cd764219ab3498b74aee7db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.937639 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 91eb4433685b4119a170054a0a777e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.938637 28321 tablet_service.cc:1467] Processing CreateTablet for tablet beba0d09f5d34980ba7ed463ff9b4c34 (DEFAULT_TABLE table=default.Table_70505af08e3d4b4aa9b097f33cae154a [id=b575b5aeba3545c6bf6d5123c857b546]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.939085 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beba0d09f5d34980ba7ed463ff9b4c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.940415 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 01482b478dcd45598fa65d14904fe276 (DEFAULT_TABLE table=default.Table_7fb65f72628b40a4855474fe07921278 [id=28d97a0a80dc4e0bbea52f5dd8e32080]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.940832 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01482b478dcd45598fa65d14904fe276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.941670 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 9cad0a4cdfa8471bbd7066b91e85022e (DEFAULT_TABLE table=default.Table_b15ac98442fe4481b0495a342ae6a57a [id=06b7f1e982194749a1caf59ad3464e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.942090 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cad0a4cdfa8471bbd7066b91e85022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.942576 28604 ts_tablet_manager.cc:1398] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 14:12:27.942874 28604 tablet_bootstrap.cc:492] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:27.944414 28604 tablet_bootstrap.cc:654] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.944530 28462 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbc326f0a9649ce9fe5976a8fb895f5 (DEFAULT_TABLE table=default.Table_d94a92f094354a9abb3e0f0173679568 [id=f187e80171a14052b5b91e3372b75d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.944934 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbc326f0a9649ce9fe5976a8fb895f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.945466 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.945854 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.946116 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 5500da0e5658492b840dd29273ab265f with delete_type TABLET_DATA_DELETED (replaced by tablet a0f4738ccaad47ab87317ee90efcd46c) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.946329 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 9cad0a4cdfa8471bbd7066b91e85022e (DEFAULT_TABLE table=default.Table_b15ac98442fe4481b0495a342ae6a57a [id=06b7f1e982194749a1caf59ad3464e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.946455 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet 5685a6b3ea6943f8a72d74de87ac2166 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:27.946692 28427 leader_election.cc:341] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.946781 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cad0a4cdfa8471bbd7066b91e85022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.947057 28427 leader_election.cc:302] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:27.907640 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 993) took 1912 ms. Trace:
I20240503 14:12:27.947515 28661 rpcz_store.cc:276] 0503 14:12:25.995035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.995505 (+ 470us) service_pool.cc:225] Handling call
0503 14:12:27.907531 (+1912026us) ts_tablet_manager.cc:1120] Deleting tablet 5de9edfb0d064939bda2ec7a9e0e500b
0503 14:12:27.907541 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.907624 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911827}
I20240503 14:12:27.947899 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 994) took 1816 ms. Trace:
I20240503 14:12:27.947969 28661 rpcz_store.cc:276] 0503 14:12:26.131474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.131547 (+ 73us) service_pool.cc:225] Handling call
0503 14:12:27.947787 (+1816240us) ts_tablet_manager.cc:1120] Deleting tablet 0d4c5648ddc84407a8d8b44f2d63dcee
0503 14:12:27.947794 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.947884 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815949}
I20240503 14:12:27.948299 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 995) took 1790 ms. Trace:
I20240503 14:12:27.948365 28661 rpcz_store.cc:276] 0503 14:12:26.157368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.157438 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:27.948191 (+1790753us) ts_tablet_manager.cc:1120] Deleting tablet 19d43cc864064c4c907ec0629391da10
0503 14:12:27.948201 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.948283 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790536}
I20240503 14:12:27.948448 28469 tablet_service.cc:1467] Processing CreateTablet for tablet adfd840fbfe7428bb9409964c120a906 (DEFAULT_TABLE table=default.Table_ddb29edceb57402988fa9a6aa01c4c1c [id=240a46975acc41fa8e8ad805b5c6f675]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.948652 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 996) took 1791 ms. Trace:
I20240503 14:12:27.948729 28661 rpcz_store.cc:276] 0503 14:12:26.157529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.157583 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:27.948557 (+1790974us) ts_tablet_manager.cc:1120] Deleting tablet 13f8553cfbab4af2a38a73e81fed2a5f
0503 14:12:27.948566 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.948639 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790854}
I20240503 14:12:27.948834 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfd840fbfe7428bb9409964c120a906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.949004 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 997) took 1790 ms. Trace:
I20240503 14:12:27.949069 28661 rpcz_store.cc:276] 0503 14:12:26.158966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.159031 (+ 65us) service_pool.cc:225] Handling call
0503 14:12:27.948908 (+1789877us) ts_tablet_manager.cc:1120] Deleting tablet 9f5b973ea16d42babd2cc75e32624854
0503 14:12:27.948917 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.948990 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1789702}
I20240503 14:12:27.949491 29996 raft_consensus.cc:2726] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.952435 28456 tablet_service.cc:1467] Processing CreateTablet for tablet ae8b8922e42449ecb0b53805a8feba5b (DEFAULT_TABLE table=default.Table_3d13d5e860f2457895d61f215a2c08cf [id=9dc6978e6751487caf5a24a481876a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.952842 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8b8922e42449ecb0b53805a8feba5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.953539 28321 tablet_service.cc:1467] Processing CreateTablet for tablet f7d18fdc987d4879a642438e748eeb62 (DEFAULT_TABLE table=default.Table_4fb75e6e800e495c9dfda4864bdebe5a [id=0b0f60024a7b467b97ed1a22e404268d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.953945 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d18fdc987d4879a642438e748eeb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.955282 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.955533 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d1314841574fc8bcf3ccb16755e421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.955669 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4846d662a64d51b16fd5447c371e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.955796 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.955914 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b41e3b21347b393497de4d4e87469 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:27.956637 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 7549b2ca31b04247bf3a2202e7585817 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:27.956737 28427 leader_election.cc:341] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:27.958153 28427 leader_election.cc:341] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:27.958685 28427 leader_election.cc:341] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:27.960498 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1006) took 1527 ms. Trace:
I20240503 14:12:27.960952 28661 rpcz_store.cc:276] 0503 14:12:26.421979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.437128 (+ 15149us) service_pool.cc:225] Handling call
0503 14:12:27.949252 (+1512124us) ts_tablet_manager.cc:1120] Deleting tablet addc2a6195034b0a927fdee7a2955122
0503 14:12:27.949261 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.949339 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511862}
I20240503 14:12:27.962018 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1007) took 1473 ms. Trace:
I20240503 14:12:27.962721 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.963053 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.963339 30142 tablet_replica.cc:330] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.963580 30142 raft_consensus.cc:2229] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.963749 30142 raft_consensus.cc:2258] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.964157 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 7549b2ca31b04247bf3a2202e7585817 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.964361 30142 ts_tablet_manager.cc:1875] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.964397 28322 tablet_service.cc:1467] Processing CreateTablet for tablet ae8b8922e42449ecb0b53805a8feba5b (DEFAULT_TABLE table=default.Table_3d13d5e860f2457895d61f215a2c08cf [id=9dc6978e6751487caf5a24a481876a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.964825 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8b8922e42449ecb0b53805a8feba5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.964752 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbc326f0a9649ce9fe5976a8fb895f5 (DEFAULT_TABLE table=default.Table_d94a92f094354a9abb3e0f0173679568 [id=f187e80171a14052b5b91e3372b75d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.968261 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.968456 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbc326f0a9649ce9fe5976a8fb895f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.968492 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 5500da0e5658492b840dd29273ab265f with delete_type TABLET_DATA_DELETED (replaced by tablet a0f4738ccaad47ab87317ee90efcd46c) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:27.963985 28337 tablet_service.cc:1467] Processing CreateTablet for tablet adfd840fbfe7428bb9409964c120a906 (DEFAULT_TABLE table=default.Table_ddb29edceb57402988fa9a6aa01c4c1c [id=240a46975acc41fa8e8ad805b5c6f675]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.968910 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfd840fbfe7428bb9409964c120a906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.971043 30142 ts_tablet_manager.cc:1888] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:27.971158 30142 log.cc:1196] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2cad43f3c6094253ba87d0ed3fc29fd9
I20240503 14:12:27.971446 30142 ts_tablet_manager.cc:1909] T 2cad43f3c6094253ba87d0ed3fc29fd9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:27.973217 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.973565 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet d93fea5caec94695a5df26d3630c50c7 with delete_type TABLET_DATA_DELETED (replaced by tablet b476e87faea1415d95ae014a93fa22af) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:27.973896 28600 tablet_bootstrap.cc:492] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:27.974040 28600 ts_tablet_manager.cc:1367] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.124s user 0.019s sys 0.011s
I20240503 14:12:27.974845 28600 raft_consensus.cc:348] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.975060 28600 raft_consensus.cc:374] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.975116 28600 raft_consensus.cc:729] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.975353 28600 consensus_queue.cc:260] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:27.975930 28600 ts_tablet_manager.cc:1398] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:27.928925 29828 ts_tablet_manager.cc:1888] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:27.976202 29828 log.cc:1196] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/3d137ef4cfc2476cbc70f1cc72ad89e6
I20240503 14:12:27.976594 29828 ts_tablet_manager.cc:1909] T 3d137ef4cfc2476cbc70f1cc72ad89e6 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:27.978065 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 971) took 2000 ms. Trace:
I20240503 14:12:27.978155 29828 rpcz_store.cc:276] 0503 14:12:25.977319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.977727 (+ 408us) service_pool.cc:225] Handling call
0503 14:12:27.852617 (+1874890us) ts_tablet_manager.cc:1120] Deleting tablet 3d137ef4cfc2476cbc70f1cc72ad89e6
0503 14:12:27.852630 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.884818 (+ 32188us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.928762 (+ 43944us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:27.978038 (+ 49276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874710}
I20240503 14:12:27.978498 29828 tablet_replica.cc:330] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:27.979110 28661 rpcz_store.cc:276] 0503 14:12:26.488512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.489661 (+ 1149us) service_pool.cc:225] Handling call
0503 14:12:27.961854 (+1472193us) ts_tablet_manager.cc:1120] Deleting tablet 77511682d71a40f5933edcb6a6523649
0503 14:12:27.961866 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.961994 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471946}
I20240503 14:12:27.979832 30253 consensus_queue.cc:1035] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.980623 30142 tablet_replica.cc:330] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:27.980904 30142 raft_consensus.cc:2229] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.981139 30142 raft_consensus.cc:2258] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:27.981729 28600 tablet_bootstrap.cc:492] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:27.983521 30253 raft_consensus.cc:482] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.983774 30253 raft_consensus.cc:504] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:27.984619 30253 leader_election.cc:288] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:27.985217 28427 consensus_peers.cc:485] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9325d5142af4b018354bf38370b61e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.985560 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1008) took 1339 ms. Trace:
I20240503 14:12:27.987112 28661 rpcz_store.cc:276] 0503 14:12:26.645946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.646009 (+ 63us) service_pool.cc:225] Handling call
0503 14:12:27.981241 (+1335232us) ts_tablet_manager.cc:1120] Deleting tablet c99c15b4d13647e98cae1375b7f13827
0503 14:12:27.981253 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.985535 (+ 4282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334969}
I20240503 14:12:27.987378 28600 tablet_bootstrap.cc:654] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.989766 30142 ts_tablet_manager.cc:1875] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:27.990746 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1009) took 1344 ms. Trace:
I20240503 14:12:27.998502 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d45788ae644a8581acca54cc55dd3f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:27.998898 28212 raft_consensus.cc:2454] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:27.978694 29828 raft_consensus.cc:2229] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:27.999819 29828 raft_consensus.cc:2258] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.000838 29828 ts_tablet_manager.cc:1875] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.002209 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e14e8b5f4046029af4673a277112d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.002787 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ac912a3bd64cdc9082596e09c72e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.003270 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d45788ae644a8581acca54cc55dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.003741 28614 tablet_bootstrap.cc:492] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.003971 28614 ts_tablet_manager.cc:1367] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.081s user 0.000s sys 0.003s
I20240503 14:12:28.004840 28614 raft_consensus.cc:348] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.005146 28614 raft_consensus.cc:374] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.005286 28614 raft_consensus.cc:729] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.005576 28614 consensus_queue.cc:260] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.006388 28614 ts_tablet_manager.cc:1398] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:28.007364 28614 tablet_bootstrap.cc:492] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.009104 28614 tablet_bootstrap.cc:654] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.009016 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd405fd84d44076b1792b7fcbe00e84 with delete_type TABLET_DATA_DELETED (replaced by tablet 1eb6d931156b492e82140ee342cb1bbb) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.009573 28661 rpcz_store.cc:276] 0503 14:12:26.646053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.646301 (+ 248us) service_pool.cc:225] Handling call
0503 14:12:27.990614 (+1344313us) ts_tablet_manager.cc:1120] Deleting tablet ed4dc732bcb44222b4b15e5d77defe33
0503 14:12:27.990628 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:27.990723 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344176}
I20240503 14:12:28.009421 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.012459 28661 tablet_replica.cc:330] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.014281 28661 raft_consensus.cc:2229] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.014422 28600 tablet_bootstrap.cc:492] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.014575 28661 raft_consensus.cc:2258] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.014694 28600 ts_tablet_manager.cc:1367] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.004s
I20240503 14:12:28.015425 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac2833a0bb045dcb2824468d936ffe4" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.015828 28600 raft_consensus.cc:348] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.016098 28600 raft_consensus.cc:374] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.016211 28600 raft_consensus.cc:729] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.016495 28600 consensus_queue.cc:260] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.016918 28352 raft_consensus.cc:3037] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.012764 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5685a6b3ea6943f8a72d74de87ac2166 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:28.012975 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 91eb4433685b4119a170054a0a777e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:28.016808 30142 ts_tablet_manager.cc:1888] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:28.012990 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:28.015542 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d1314841574fc8bcf3ccb16755e421" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:28.017326 30142 log.cc:1196] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/9ac2833a0bb045dcb2824468d936ffe4
I20240503 14:12:28.017679 30142 ts_tablet_manager.cc:1909] T 9ac2833a0bb045dcb2824468d936ffe4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:28.016073 28604 tablet_bootstrap.cc:492] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.017904 28604 ts_tablet_manager.cc:1367] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.075s user 0.006s sys 0.000s
I20240503 14:12:28.018134 28600 ts_tablet_manager.cc:1398] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:28.018405 28661 ts_tablet_manager.cc:1875] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.020879 30266 raft_consensus.cc:482] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.021042 30266 raft_consensus.cc:504] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.021684 30266 leader_election.cc:288] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.023193 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.023262 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:28.023911 28150 leader_election.cc:341] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:28.024238 28150 leader_election.cc:341] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.024399 28150 leader_election.cc:302] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.024542 28600 tablet_bootstrap.cc:492] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.025102 30266 raft_consensus.cc:2726] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.025959 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 91eb4433685b4119a170054a0a777e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:28.026712 28600 tablet_bootstrap.cc:654] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.028370 30142 tablet_replica.cc:330] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:28.028573 30142 raft_consensus.cc:2229] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.028748 30142 raft_consensus.cc:2258] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.029345 30142 ts_tablet_manager.cc:1875] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.029608 29828 ts_tablet_manager.cc:1888] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.030550 28604 raft_consensus.cc:348] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.030874 28604 raft_consensus.cc:374] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.030979 28604 raft_consensus.cc:729] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.031365 28604 consensus_queue.cc:260] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.032541 28604 ts_tablet_manager.cc:1398] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:12:28.032946 28604 tablet_bootstrap.cc:492] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.032932 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 3d54cc5c03034a6685c1da34f9bd65ac (table default.Table_8f505fa519514278ae7489b9ff7bbaf2 [id=f58e9b32e96948cfab2760b64eb18b68]) successfully deleted
W20240503 14:12:28.033308 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 8c6202333af943cbbc0d09553c1c193c because the tablet was not found. No further retry: Not found: Tablet not found: 8c6202333af943cbbc0d09553c1c193c
W20240503 14:12:28.033658 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet addc2a6195034b0a927fdee7a2955122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet addc2a6195034b0a927fdee7a2955122 already in progress: creating tablet
W20240503 14:12:28.034018 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 77511682d71a40f5933edcb6a6523649 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77511682d71a40f5933edcb6a6523649 already in progress: creating tablet
W20240503 14:12:28.034346 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c99c15b4d13647e98cae1375b7f13827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c99c15b4d13647e98cae1375b7f13827 already in progress: creating tablet
W20240503 14:12:28.034621 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d2eb8328e86a4ee5a53647a810729519 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2eb8328e86a4ee5a53647a810729519 already in progress: creating tablet
W20240503 14:12:28.034893 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7b7ca19dd15f48a68ace7fefc7ae660f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b7ca19dd15f48a68ace7fefc7ae660f already in progress: creating tablet
I20240503 14:12:28.036362 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]) successfully deleted
I20240503 14:12:28.036494 30142 ts_tablet_manager.cc:1888] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:28.036643 30142 log.cc:1196] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/11e14e8b5f4046029af4673a277112d9
W20240503 14:12:28.036773 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7d348da459634561985c4ac1710c0bb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d348da459634561985c4ac1710c0bb2 already in progress: creating tablet
I20240503 14:12:28.036996 30142 ts_tablet_manager.cc:1909] T 11e14e8b5f4046029af4673a277112d9 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:28.039064 29996 raft_consensus.cc:482] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.039256 29996 raft_consensus.cc:504] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.039978 29996 leader_election.cc:288] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.040416 28427 leader_election.cc:302] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.041029 29996 raft_consensus.cc:2781] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.041127 29996 raft_consensus.cc:482] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.041200 29996 raft_consensus.cc:3037] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.043977 29996 raft_consensus.cc:504] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.044302 30266 raft_consensus.cc:482] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.044474 30266 raft_consensus.cc:504] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.044975 29996 leader_election.cc:288] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:28.045130 30266 leader_election.cc:288] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:28.046554 28427 leader_election.cc:341] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:28.046694 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49445f0f1dd1451c9ba066e61273f2f1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:28.046900 28427 consensus_peers.cc:485] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e8411f474c247c2abfd0bd9682fb8a2. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:28.047091 28427 consensus_peers.cc:485] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b90e9563b4c4cef8ee7964eaa96e285. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:28.047260 28150 leader_election.cc:341] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.048316 28604 tablet_bootstrap.cc:654] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.049206 29828 log.cc:1196] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/dcd405fd84d44076b1792b7fcbe00e84
I20240503 14:12:28.049793 30142 tablet_replica.cc:330] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:28.048043 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d45788ae644a8581acca54cc55dd3f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.046504 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49445f0f1dd1451c9ba066e61273f2f1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.052539 29828 ts_tablet_manager.cc:1909] T dcd405fd84d44076b1792b7fcbe00e84 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:28.052862 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 972) took 2074 ms. Trace:
I20240503 14:12:28.051546 28614 tablet_bootstrap.cc:492] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.053280 28614 ts_tablet_manager.cc:1367] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.003s
I20240503 14:12:28.054177 28614 raft_consensus.cc:348] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.054466 28614 raft_consensus.cc:374] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.054564 28614 raft_consensus.cc:729] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.054867 28614 consensus_queue.cc:260] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:28.053099 28150 leader_election.cc:341] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.055188 28150 leader_election.cc:302] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.055462 30266 raft_consensus.cc:2726] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.050024 30142 raft_consensus.cc:2229] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.055851 30142 raft_consensus.cc:2258] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.056147 28614 ts_tablet_manager.cc:1398] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:28.056535 28614 tablet_bootstrap.cc:492] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.056511 30142 ts_tablet_manager.cc:1875] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.057924 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.052953 29828 rpcz_store.cc:276] 0503 14:12:25.978518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.978574 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:27.978447 (+1999873us) ts_tablet_manager.cc:1120] Deleting tablet dcd405fd84d44076b1792b7fcbe00e84
0503 14:12:27.978457 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.019676 (+ 41219us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.029448 (+ 9772us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.052843 (+ 23395us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999710}
I20240503 14:12:28.058187 28213 raft_consensus.cc:2454] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.058462 29828 tablet_replica.cc:330] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:28.059540 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d45788ae644a8581acca54cc55dd3f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:28.060300 29828 raft_consensus.cc:2229] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.062050 29828 raft_consensus.cc:2258] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.063591 29828 ts_tablet_manager.cc:1875] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.058518 28614 tablet_bootstrap.cc:654] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:28.066886 28427 consensus_peers.cc:485] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d54cc5c03034a6685c1da34f9bd65ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.067111 28427 leader_election.cc:302] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.069936 30142 ts_tablet_manager.cc:1888] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:28.070042 28661 ts_tablet_manager.cc:1888] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.070101 30142 log.cc:1196] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/02d45788ae644a8581acca54cc55dd3f
I20240503 14:12:28.070145 28661 log.cc:1196] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/11ac912a3bd64cdc9082596e09c72e13
I20240503 14:12:28.070649 29994 raft_consensus.cc:2781] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.070796 29994 raft_consensus.cc:482] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.070942 29994 raft_consensus.cc:3037] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.071808 30335 raft_consensus.cc:482] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.071980 30335 raft_consensus.cc:504] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.072764 30335 leader_election.cc:288] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.073372 30263 raft_consensus.cc:482] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.073534 30263 raft_consensus.cc:504] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.073940 29994 raft_consensus.cc:504] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.074230 30263 leader_election.cc:288] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.075877 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac2833a0bb045dcb2824468d936ffe4" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.076355 29994 leader_election.cc:288] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:28.076784 28427 leader_election.cc:341] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: Tablet not RUNNING: STOPPED
I20240503 14:12:28.078040 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b53efefcf42569bba2fcabd21e211" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.079033 30142 ts_tablet_manager.cc:1909] T 02d45788ae644a8581acca54cc55dd3f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:28.078791 28213 raft_consensus.cc:2454] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.080471 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:28.082903 28212 raft_consensus.cc:3037] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.086014 28212 raft_consensus.cc:2454] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:28.086678 28661 ts_tablet_manager.cc:1909] T 11ac912a3bd64cdc9082596e09c72e13 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.088920 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.089999 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d45788ae644a8581acca54cc55dd3f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:28.037041 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 728ba3ac1cb645168481b84b5b548140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728ba3ac1cb645168481b84b5b548140 already in progress: creating tablet
I20240503 14:12:28.091807 29828 ts_tablet_manager.cc:1888] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:28.091128 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 54106.7us
W20240503 14:12:28.093031 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3cbcae22ea124edaa14c9409f3052714 because the tablet was not found. No further retry: Not found: Tablet not found: 3cbcae22ea124edaa14c9409f3052714
I20240503 14:12:28.097718 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1010) took 1451 ms. Trace:
I20240503 14:12:28.097972 28661 rpcz_store.cc:276] 0503 14:12:26.647438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.647498 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:28.012387 (+1364889us) ts_tablet_manager.cc:1120] Deleting tablet 11ac912a3bd64cdc9082596e09c72e13
0503 14:12:28.012400 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.021564 (+ 9164us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.069939 (+ 48375us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.097685 (+ 27746us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25,"tablet-delete.queue_time_us":1364665}
I20240503 14:12:28.098543 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1012) took 1450 ms. Trace:
I20240503 14:12:28.098707 28661 rpcz_store.cc:276] 0503 14:12:26.647672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.647770 (+ 98us) service_pool.cc:225] Handling call
0503 14:12:28.098435 (+1450665us) ts_tablet_manager.cc:1120] Deleting tablet 495190fca06a40a386c8215eb2195920
0503 14:12:28.098446 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.098522 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450541}
W20240503 14:12:28.098979 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 4084bd4fbf604042bb33ff03c3f8cc19 because the tablet was not found. No further retry: Not found: Tablet not found: 4084bd4fbf604042bb33ff03c3f8cc19
I20240503 14:12:28.099107 28661 tablet_replica.cc:330] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.099355 28661 raft_consensus.cc:2229] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.099786 28661 raft_consensus.cc:2258] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.100206 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884cf4ea11e48ef941722b79ad50075" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.100678 28661 ts_tablet_manager.cc:1875] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.096598 28427 leader_election.cc:302] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.103711 28427 leader_election.cc:302] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.103922 28600 tablet_bootstrap.cc:492] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.104208 28600 ts_tablet_manager.cc:1367] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.080s user 0.020s sys 0.005s
I20240503 14:12:28.104225 30335 raft_consensus.cc:2781] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.104364 30335 raft_consensus.cc:686] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:28.103979 29994 raft_consensus.cc:2781] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.104686 29994 raft_consensus.cc:482] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.104784 29994 raft_consensus.cc:3037] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.104650 30335 consensus_queue.cc:237] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.105494 28600 raft_consensus.cc:348] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.105676 28600 raft_consensus.cc:374] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.105720 28600 raft_consensus.cc:729] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.105966 28600 consensus_queue.cc:260] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.102911 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet dbcc6cef7b7d464a9fbbae61caf0437d (table default.Table_7c4e797497e74bbe98ff720b9ffd05ff [id=ee4e5312119d414aa8f8d01c28e0b5fb]) successfully deleted
I20240503 14:12:28.107921 28604 tablet_bootstrap.cc:492] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.107961 29994 raft_consensus.cc:504] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.108741 28604 ts_tablet_manager.cc:1367] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.002s
I20240503 14:12:28.109308 28600 ts_tablet_manager.cc:1398] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:28.109289 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a884cf4ea11e48ef941722b79ad50075 (table default.Table_cc77503fb1ad4d03a6278aef9f3ab979 [id=4b3b8bed2fba4370a246902bec7554f7]) successfully deleted
I20240503 14:12:28.109557 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet c9325d5142af4b018354bf38370b61e0 (table default.Table_2ce3b6f2b77b4607856fc2a64ead5bb5 [id=db393a7e2a83466385ffc0af3d2789c4]) successfully deleted
I20240503 14:12:28.109694 28604 raft_consensus.cc:348] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.109884 28600 tablet_bootstrap.cc:492] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.111187 28604 raft_consensus.cc:374] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.111189 30318 tablet.cc:1722] T 77c77a11ef384709aef5b62273380573 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:28.111471 28604 raft_consensus.cc:729] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.111758 28604 consensus_queue.cc:260] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.111968 28600 tablet_bootstrap.cc:654] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.112365 28604 ts_tablet_manager.cc:1398] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:28.113704 28604 tablet_bootstrap.cc:492] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.116367 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e0175b38c1e0464a9a6208cfaf68ca0d (table default.Table_7fdf02629122402c9cbf8053f89eced3 [id=5201e94e1d164116b74d4329e5991eca]) successfully deleted
I20240503 14:12:28.117893 29994 leader_election.cc:288] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:28.118196 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet bab214b7ec58425d8cd1180ee16b273d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab214b7ec58425d8cd1180ee16b273d already in progress: creating tablet
W20240503 14:12:28.118502 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c725628f72884086b523b56ecf876069 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c725628f72884086b523b56ecf876069 already in progress: creating tablet
I20240503 14:12:28.118852 28604 tablet_bootstrap.cc:654] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.119910 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b53efefcf42569bba2fcabd21e211" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:28.120268 28212 raft_consensus.cc:3037] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.121256 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.121527 28213 raft_consensus.cc:2379] T 42c3868ce2744d349a23779b2a62bb32 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:28.123909 28212 raft_consensus.cc:2454] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:28.133349 30266 raft_consensus.cc:482] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.133538 30266 raft_consensus.cc:504] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.134336 30266 leader_election.cc:288] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.092401 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b53efefcf42569bba2fcabd21e211" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.096386 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:28.102471 28614 tablet_bootstrap.cc:492] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.128247 30263 consensus_queue.cc:1035] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.130273 30312 consensus_queue.cc:1035] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.135965 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.136847 28614 ts_tablet_manager.cc:1367] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.003s
I20240503 14:12:28.137562 29828 log.cc:1196] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/4084bd4fbf604042bb33ff03c3f8cc19
W20240503 14:12:28.137527 28150 leader_election.cc:341] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.138001 29828 ts_tablet_manager.cc:1909] T 4084bd4fbf604042bb33ff03c3f8cc19 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:28.138350 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 973) took 2159 ms. Trace:
I20240503 14:12:28.121697 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c3868ce2744d349a23779b2a62bb32" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:28.130100 28427 leader_election.cc:341] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.140257 28427 leader_election.cc:302] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:28.135318 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.140997 29994 raft_consensus.cc:2726] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.141042 30335 consensus_queue.cc:1035] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.141873 29996 consensus_queue.cc:1035] T a703cfbaad2a4a618abb9e07b4930cee P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:28.142207 28150 leader_election.cc:341] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.142360 28150 leader_election.cc:302] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.142889 30266 raft_consensus.cc:2726] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:28.143759 28427 leader_election.cc:341] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:28.144235 28427 leader_election.cc:341] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.144392 28427 leader_election.cc:302] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:28.144685 30335 raft_consensus.cc:2726] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:28.144961 28427 leader_election.cc:341] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:28.145291 28427 leader_election.cc:341] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.145397 28427 leader_election.cc:302] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:28.145613 30335 raft_consensus.cc:2726] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.149084 29828 rpcz_store.cc:276] 0503 14:12:25.978874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.979491 (+ 617us) service_pool.cc:225] Handling call
0503 14:12:28.058405 (+2078914us) ts_tablet_manager.cc:1120] Deleting tablet 4084bd4fbf604042bb33ff03c3f8cc19
0503 14:12:28.058416 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.088691 (+ 30275us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.091685 (+ 2994us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.138326 (+ 46641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2078749}
I20240503 14:12:28.150463 28427 leader_election.cc:302] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.144049 28614 raft_consensus.cc:348] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.150817 30253 raft_consensus.cc:2781] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.150841 28614 raft_consensus.cc:374] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.150941 30253 raft_consensus.cc:686] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:28.150949 28614 raft_consensus.cc:729] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.151219 30253 consensus_queue.cc:237] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.151199 28614 consensus_queue.cc:260] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.151984 28614 ts_tablet_manager.cc:1398] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 14:12:28.152400 28614 tablet_bootstrap.cc:492] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.153944 28614 tablet_bootstrap.cc:654] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.155721 30272 raft_consensus.cc:504] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.157070 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de9e159d2ba47039dd6bd207a9cddeb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:28.157799 28150 leader_election.cc:341] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.156356 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 974) took 2172 ms. Trace:
I20240503 14:12:28.158248 29828 rpcz_store.cc:276] 0503 14:12:25.984373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:25.989041 (+ 4668us) service_pool.cc:225] Handling call
0503 14:12:28.156203 (+2167162us) ts_tablet_manager.cc:1120] Deleting tablet 8410e8b6b511403ea609b1353855937b
0503 14:12:28.156218 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.156330 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2166940}
I20240503 14:12:28.158573 29828 tablet_replica.cc:330] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:28.158787 29828 raft_consensus.cc:2229] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.158999 29828 raft_consensus.cc:2258] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.159688 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b53efefcf42569bba2fcabd21e211" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:28.164283 28427 leader_election.cc:341] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:28.164734 28427 leader_election.cc:341] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.167157 28600 tablet_bootstrap.cc:492] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.167433 28600 ts_tablet_manager.cc:1367] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.022s
I20240503 14:12:28.169107 28600 raft_consensus.cc:348] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.169633 28600 raft_consensus.cc:374] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.169764 28600 raft_consensus.cc:729] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.170130 28600 consensus_queue.cc:260] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.170783 28600 ts_tablet_manager.cc:1398] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20240503 14:12:28.171932 28427 consensus_peers.cc:485] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0fd8fb8ea1a4fc3bfa4f55ea59ae12a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.138160 28661 ts_tablet_manager.cc:1888] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.172176 28661 log.cc:1196] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/57ab077988364bd687e54f90c3c265d5
I20240503 14:12:28.172617 28661 ts_tablet_manager.cc:1909] T 57ab077988364bd687e54f90c3c265d5 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.173978 28600 tablet_bootstrap.cc:492] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.177088 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1014) took 1529 ms. Trace:
I20240503 14:12:28.178694 28600 tablet_bootstrap.cc:654] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:28.182054 28290 leader_election.cc:341] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a884cf4ea11e48ef941722b79ad50075
I20240503 14:12:28.182236 28290 leader_election.cc:302] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.184391 30315 raft_consensus.cc:2726] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.185016 28661 rpcz_store.cc:276] 0503 14:12:26.647844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.647911 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:28.099052 (+1451141us) ts_tablet_manager.cc:1120] Deleting tablet 57ab077988364bd687e54f90c3c265d5
0503 14:12:28.099062 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.131784 (+ 32722us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.138006 (+ 6222us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.177051 (+ 39045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451039}
I20240503 14:12:28.186117 28661 tablet_replica.cc:330] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.186425 28661 raft_consensus.cc:2229] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.186892 28661 raft_consensus.cc:2258] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.188606 30315 consensus_queue.cc:1035] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.189484 29828 ts_tablet_manager.cc:1875] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.195048 28290 leader_election.cc:302] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.196815 30253 consensus_queue.cc:1035] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.197863 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de9e159d2ba47039dd6bd207a9cddeb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.197604 28661 ts_tablet_manager.cc:1875] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:28.198501 28150 leader_election.cc:341] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.198640 28150 leader_election.cc:302] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.198884 30266 raft_consensus.cc:2726] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.205601 30315 raft_consensus.cc:3037] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.208444 30315 raft_consensus.cc:2726] T 42c3868ce2744d349a23779b2a62bb32 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.217104 30272 leader_election.cc:288] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.204613 28604 tablet_bootstrap.cc:492] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.220387 28604 ts_tablet_manager.cc:1367] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.000s
W20240503 14:12:28.220614 28427 leader_election.cc:341] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.221485 28604 raft_consensus.cc:348] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.221796 28604 raft_consensus.cc:374] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.221895 28604 raft_consensus.cc:729] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.222213 28604 consensus_queue.cc:260] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.223598 29996 raft_consensus.cc:482] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.223738 29996 raft_consensus.cc:504] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.224413 29996 leader_election.cc:288] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:28.225330 28661 ts_tablet_manager.cc:1888] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.228125 30315 raft_consensus.cc:482] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.228405 30315 raft_consensus.cc:504] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.229489 30315 leader_election.cc:288] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.230448 28661 log.cc:1196] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
W20240503 14:12:28.231180 28427 leader_election.cc:341] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:28.233901 28661 ts_tablet_manager.cc:1909] T f0fd8fb8ea1a4fc3bfa4f55ea59ae12a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.237737 28604 ts_tablet_manager.cc:1398] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 14:12:28.243337 29828 ts_tablet_manager.cc:1888] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.245172 28604 tablet_bootstrap.cc:492] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.246899 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1011) took 1599 ms. Trace:
I20240503 14:12:28.247089 28661 rpcz_store.cc:276] 0503 14:12:26.647589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.648027 (+ 438us) service_pool.cc:225] Handling call
0503 14:12:28.185578 (+1537551us) ts_tablet_manager.cc:1120] Deleting tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
0503 14:12:28.186034 (+ 456us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.221844 (+ 35810us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.225193 (+ 3349us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.246868 (+ 21675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1537411}
I20240503 14:12:28.247400 28604 tablet_bootstrap.cc:654] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.247666 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1013) took 1599 ms. Trace:
I20240503 14:12:28.247988 28661 rpcz_store.cc:276] 0503 14:12:26.647755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.648176 (+ 421us) service_pool.cc:225] Handling call
0503 14:12:28.247511 (+1599335us) ts_tablet_manager.cc:1120] Deleting tablet dda39f0323ec47799d3d44c96f3ba605
0503 14:12:28.247522 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.247646 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599217}
I20240503 14:12:28.248510 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1015) took 1593 ms. Trace:
I20240503 14:12:28.248785 28661 rpcz_store.cc:276] 0503 14:12:26.655516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.655591 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:28.248379 (+1592788us) ts_tablet_manager.cc:1120] Deleting tablet e4a544fc27624dcc8a4cb003e59408f2
0503 14:12:28.248390 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.248491 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592578}
I20240503 14:12:28.249540 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1017) took 1593 ms. Trace:
I20240503 14:12:28.249682 28661 rpcz_store.cc:276] 0503 14:12:26.655739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.655838 (+ 99us) service_pool.cc:225] Handling call
0503 14:12:28.249151 (+1593313us) ts_tablet_manager.cc:1120] Deleting tablet 7b7ca19dd15f48a68ace7fefc7ae660f
0503 14:12:28.249397 (+ 246us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.249521 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593192}
I20240503 14:12:28.253296 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1019) took 1597 ms. Trace:
I20240503 14:12:28.253511 28661 rpcz_store.cc:276] 0503 14:12:26.655908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.655981 (+ 73us) service_pool.cc:225] Handling call
0503 14:12:28.250230 (+1594249us) ts_tablet_manager.cc:1120] Deleting tablet 7d348da459634561985c4ac1710c0bb2
0503 14:12:28.253125 (+ 2895us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.253264 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1594149}
I20240503 14:12:28.253630 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816f3b05cfb4496984f6cb28d310cc4" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.253643 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816f3b05cfb4496984f6cb28d310cc4" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.253990 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1016) took 1598 ms. Trace:
I20240503 14:12:28.254068 28493 raft_consensus.cc:2379] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate a64bc15d65d84ad5a005960a9a84df9c in this term.
I20240503 14:12:28.254324 28661 rpcz_store.cc:276] 0503 14:12:26.655655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.656124 (+ 469us) service_pool.cc:225] Handling call
0503 14:12:28.253856 (+1597732us) ts_tablet_manager.cc:1120] Deleting tablet d2eb8328e86a4ee5a53647a810729519
0503 14:12:28.253866 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.253959 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597610}
I20240503 14:12:28.254806 28661 tablet_replica.cc:330] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.255224 28661 raft_consensus.cc:2229] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.255934 28661 raft_consensus.cc:2258] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.256729 28661 ts_tablet_manager.cc:1875] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.258821 30253 consensus_queue.cc:1035] T 42c3868ce2744d349a23779b2a62bb32 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.268251 29996 raft_consensus.cc:482] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.268494 29996 raft_consensus.cc:504] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.269265 29996 leader_election.cc:288] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:28.271853 28427 leader_election.cc:341] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 13c7a51c68ef478e9bdc5b3e3dc3d5df
W20240503 14:12:28.274505 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 156011us
I20240503 14:12:28.277163 28600 tablet_bootstrap.cc:492] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.277408 28600 ts_tablet_manager.cc:1367] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.104s user 0.019s sys 0.016s
I20240503 14:12:28.278285 28600 raft_consensus.cc:348] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.278563 28600 raft_consensus.cc:374] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.278633 28600 raft_consensus.cc:729] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.278934 28600 consensus_queue.cc:260] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.279541 28600 ts_tablet_manager.cc:1398] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:28.280335 28600 tablet_bootstrap.cc:492] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.282269 28600 tablet_bootstrap.cc:654] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:28.284217 30103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:28.284782 30103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:28.285053 30103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:28.293546 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ce4846d662a64d51b16fd5447c371e9a (table default.Table_4fdbe31ba57a42b993d60fa19081c044 [id=ea4ef9402edd4db9820a6a318bfec37d]) successfully deleted
I20240503 14:12:28.293826 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a3d1314841574fc8bcf3ccb16755e421 (table default.Table_ddd5f026bb204de99edef3cc54bd8347 [id=e5c443cd73d340e99806c89d5b8e659f]) successfully deleted
I20240503 14:12:28.293969 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 62e4f6e3908e420e9fa156ee24b437d6 (table default.Table_5654996201694c1b90fc6e728e068ca3 [id=9a6689ece91646069be1867e8f141392]) successfully deleted
I20240503 14:12:28.294090 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df (table default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]) successfully deleted
W20240503 14:12:28.294282 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5ac145bcc44b4437b840df389cd254f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ac145bcc44b4437b840df389cd254f9 already in progress: creating tablet
W20240503 14:12:28.294445 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9db657eef2db4e3791318e01e1a965fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9db657eef2db4e3791318e01e1a965fb already in progress: creating tablet
I20240503 14:12:28.294700 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 920b41e3b21347b393497de4d4e87469 (table default.Table_28c16c4a952d45c0b34fd1d4864ac748 [id=2fd597e69c1041a2984ca535058f6fab]) successfully deleted
W20240503 14:12:28.294873 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e04c0b6955b34267820ceecfffcfb88a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e04c0b6955b34267820ceecfffcfb88a already in progress: creating tablet
W20240503 14:12:28.295018 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a526e9f1e9934e519349dcd4523ee694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a526e9f1e9934e519349dcd4523ee694 already in progress: creating tablet
W20240503 14:12:28.295159 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 57ab077988364bd687e54f90c3c265d5 because the tablet was not found. No further retry: Not found: Tablet not found: 57ab077988364bd687e54f90c3c265d5
W20240503 14:12:28.295321 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 because the tablet was not found. No further retry: Not found: Tablet not found: 3d137ef4cfc2476cbc70f1cc72ad89e6
I20240503 14:12:28.295542 28661 ts_tablet_manager.cc:1888] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:28.244458 28614 tablet_bootstrap.cc:492] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.246502 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e4f6e3908e420e9fa156ee24b437d6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.296311 28614 ts_tablet_manager.cc:1367] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.144s user 0.003s sys 0.001s
I20240503 14:12:28.292826 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c7a51c68ef478e9bdc5b3e3dc3d5df" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.297307 28614 raft_consensus.cc:348] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.297529 28614 raft_consensus.cc:374] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.297654 28614 raft_consensus.cc:729] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.297931 28614 consensus_queue.cc:260] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.298660 28614 ts_tablet_manager.cc:1398] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:28.299010 28614 tablet_bootstrap.cc:492] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.300971 28614 tablet_bootstrap.cc:654] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.301302 28661 log.cc:1196] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/5b90e9563b4c4cef8ee7964eaa96e285
I20240503 14:12:28.303525 29828 log.cc:1196] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/8c6202333af943cbbc0d09553c1c193c
I20240503 14:12:28.303930 29828 ts_tablet_manager.cc:1909] T 8c6202333af943cbbc0d09553c1c193c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:28.304368 28661 ts_tablet_manager.cc:1909] T 5b90e9563b4c4cef8ee7964eaa96e285 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.304474 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 975) took 2314 ms. Trace:
I20240503 14:12:28.304644 29828 rpcz_store.cc:276] 0503 14:12:25.989545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.015832 (+ 26287us) service_pool.cc:225] Handling call
0503 14:12:28.158517 (+2142685us) ts_tablet_manager.cc:1120] Deleting tablet 8c6202333af943cbbc0d09553c1c193c
0503 14:12:28.158528 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.240189 (+ 81661us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.243207 (+ 3018us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.304444 (+ 61237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2142462}
I20240503 14:12:28.305208 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 976) took 2288 ms. Trace:
I20240503 14:12:28.305538 29828 rpcz_store.cc:276] 0503 14:12:26.016280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.016340 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:28.305085 (+2288745us) ts_tablet_manager.cc:1120] Deleting tablet bf21f6234dd64cf8a32f1599fe50746e
0503 14:12:28.305095 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.305190 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288557}
I20240503 14:12:28.306649 29828 tablet_replica.cc:330] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:28.306725 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1018) took 1650 ms. Trace:
I20240503 14:12:28.306938 29828 raft_consensus.cc:2229] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.307180 29828 raft_consensus.cc:2258] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.312279 28661 rpcz_store.cc:276] 0503 14:12:26.655826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.656304 (+ 478us) service_pool.cc:225] Handling call
0503 14:12:28.254704 (+1598400us) ts_tablet_manager.cc:1120] Deleting tablet 5b90e9563b4c4cef8ee7964eaa96e285
0503 14:12:28.254714 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.266507 (+ 11793us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.295327 (+ 28820us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.306680 (+ 11353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598288}
I20240503 14:12:28.303396 28604 tablet_bootstrap.cc:492] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.316311 28604 ts_tablet_manager.cc:1367] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.002s
I20240503 14:12:28.317603 28604 raft_consensus.cc:348] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.318224 28604 raft_consensus.cc:374] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.313979 28347 raft_consensus.cc:3037] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.318130 28290 leader_election.cc:302] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.320466 30315 raft_consensus.cc:3037] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.322571 29828 ts_tablet_manager.cc:1875] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:28.325521 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b97da778da214bb6bc147c5ada6c7603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b97da778da214bb6bc147c5ada6c7603 already in progress: creating tablet
W20240503 14:12:28.324128 28427 leader_election.cc:341] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.326673 28427 leader_election.cc:302] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:28.327821 28427 leader_election.cc:341] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.328002 28427 leader_election.cc:302] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:28.322922 30315 raft_consensus.cc:2726] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.328398 29996 raft_consensus.cc:2726] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.328625 29996 raft_consensus.cc:2726] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:28.331315 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f91ddbc250c44fbfbf4ffe83433d8dd8 already in progress: creating tablet
W20240503 14:12:28.332553 30103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:28.333335 30103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20240503 14:12:28.333534 28614 tablet_bootstrap.cc:492] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.333693 28614 ts_tablet_manager.cc:1367] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.000s
I20240503 14:12:28.333711 29996 consensus_queue.cc:1035] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:28.333906 30103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:28.334358 30103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:28.334717 28614 raft_consensus.cc:348] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.334937 28614 raft_consensus.cc:374] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.334995 28614 raft_consensus.cc:729] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.335419 28347 raft_consensus.cc:1226] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Rejecting Update request from peer a64bc15d65d84ad5a005960a9a84df9c for earlier term 1. Current term is 2. Ops: []
I20240503 14:12:28.335887 28614 consensus_queue.cc:260] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.336237 30272 consensus_queue.cc:1046] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:28.336560 28614 ts_tablet_manager.cc:1398] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:28.336717 30272 raft_consensus.cc:3032] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Stepping down as leader of term 1
W20240503 14:12:28.336722 28290 consensus_peers.cc:485] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0175b38c1e0464a9a6208cfaf68ca0d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.336799 30272 raft_consensus.cc:729] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:28.336947 28614 tablet_bootstrap.cc:492] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.337152 30272 consensus_queue.cc:260] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.337653 30272 raft_consensus.cc:3037] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:28.339069 28614 tablet_bootstrap.cc:654] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:28.346073 28427 consensus_peers.cc:485] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a884cf4ea11e48ef941722b79ad50075. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.346892 28604 raft_consensus.cc:729] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.347393 28604 consensus_queue.cc:260] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.313005 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1022) took 1489 ms. Trace:
I20240503 14:12:28.353286 28661 rpcz_store.cc:276] 0503 14:12:26.823617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.823690 (+ 73us) service_pool.cc:225] Handling call
0503 14:12:28.312772 (+1489082us) ts_tablet_manager.cc:1120] Deleting tablet bab214b7ec58425d8cd1180ee16b273d
0503 14:12:28.312788 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.312926 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1488842}
I20240503 14:12:28.353986 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1020) took 1530 ms. Trace:
I20240503 14:12:28.354218 28661 rpcz_store.cc:276] 0503 14:12:26.823407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.823961 (+ 554us) service_pool.cc:225] Handling call
0503 14:12:28.353719 (+1529758us) ts_tablet_manager.cc:1120] Deleting tablet 728ba3ac1cb645168481b84b5b548140
0503 14:12:28.353734 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.353908 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529636}
I20240503 14:12:28.354714 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1023) took 1530 ms. Trace:
I20240503 14:12:28.354877 28661 rpcz_store.cc:276] 0503 14:12:26.823709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.827641 (+ 3932us) service_pool.cc:225] Handling call
0503 14:12:28.354518 (+1526877us) ts_tablet_manager.cc:1120] Deleting tablet c725628f72884086b523b56ecf876069
0503 14:12:28.354530 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.354658 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526688}
I20240503 14:12:28.355301 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1033) took 1524 ms. Trace:
I20240503 14:12:28.360464 28661 rpcz_store.cc:276] 0503 14:12:26.830800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.830850 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:28.355146 (+1524296us) ts_tablet_manager.cc:1120] Deleting tablet ae1cd28bc07e4ccd9097f6dee471c5c6
0503 14:12:28.355157 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.355251 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524117}
I20240503 14:12:28.361335 28600 tablet_bootstrap.cc:492] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.361591 28600 ts_tablet_manager.cc:1367] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.081s user 0.017s sys 0.013s
I20240503 14:12:28.356824 28604 ts_tablet_manager.cc:1398] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.040s user 0.002s sys 0.001s
I20240503 14:12:28.362416 28604 tablet_bootstrap.cc:492] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.363107 28600 raft_consensus.cc:348] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.363423 28600 raft_consensus.cc:374] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.363540 28600 raft_consensus.cc:729] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.363656 28614 tablet_bootstrap.cc:492] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.363894 28614 ts_tablet_manager.cc:1367] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.006s
I20240503 14:12:28.363878 28600 consensus_queue.cc:260] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.365017 28614 raft_consensus.cc:348] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.365473 28614 raft_consensus.cc:374] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.365679 28614 raft_consensus.cc:729] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.366111 28614 consensus_queue.cc:260] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.366978 28600 ts_tablet_manager.cc:1398] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:28.367596 28600 tablet_bootstrap.cc:492] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.367795 28661 tablet_replica.cc:330] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.368294 28614 ts_tablet_manager.cc:1398] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:28.368871 28614 tablet_bootstrap.cc:492] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.369752 28600 tablet_bootstrap.cc:654] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.375939 28614 tablet_bootstrap.cc:654] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.380854 30103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.50.212:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.27.50.254:36571,127.27.50.253:44259,127.27.50.252:45363
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.50.254:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.50.254
--webserver_port=38435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:28.385885 29828 ts_tablet_manager.cc:1888] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.386250 28600 tablet_bootstrap.cc:492] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.386518 28600 ts_tablet_manager.cc:1367] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240503 14:12:28.387485 29828 log.cc:1196] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/d43823da09bb49be8f5d1a1149a4b11d
I20240503 14:12:28.387415 28600 raft_consensus.cc:348] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.387638 28600 raft_consensus.cc:374] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.387697 28600 raft_consensus.cc:729] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.387957 28600 consensus_queue.cc:260] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.388581 29828 ts_tablet_manager.cc:1909] T d43823da09bb49be8f5d1a1149a4b11d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:28.389941 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 977) took 2373 ms. Trace:
I20240503 14:12:28.392844 28600 ts_tablet_manager.cc:1398] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:28.393177 28600 tablet_bootstrap.cc:492] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.394915 30103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20240503 14:12:28.394912 28427 consensus_peers.cc:485] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce4846d662a64d51b16fd5447c371e9a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.388953 28614 tablet_bootstrap.cc:492] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.396502 28614 ts_tablet_manager.cc:1367] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.004s
I20240503 14:12:28.397540 28614 raft_consensus.cc:348] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.398355 28614 raft_consensus.cc:374] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.398548 28614 raft_consensus.cc:729] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.398825 28614 consensus_queue.cc:260] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.399757 28614 ts_tablet_manager.cc:1398] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:28.400462 28614 tablet_bootstrap.cc:492] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.401567 28600 tablet_bootstrap.cc:654] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.402487 28614 tablet_bootstrap.cc:654] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.403034 30103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:28.403741 28661 raft_consensus.cc:2229] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.402602 29828 rpcz_store.cc:276] 0503 14:12:26.016679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.016720 (+ 41us) service_pool.cc:225] Handling call
0503 14:12:28.306590 (+2289870us) ts_tablet_manager.cc:1120] Deleting tablet d43823da09bb49be8f5d1a1149a4b11d
0503 14:12:28.306601 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.327969 (+ 21368us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.373225 (+ 45256us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.389912 (+ 16687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2289723}
I20240503 14:12:28.368494 28604 tablet_bootstrap.cc:654] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.406778 29828 tablet_replica.cc:330] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:28.407166 29828 raft_consensus.cc:2229] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.407405 29828 raft_consensus.cc:2258] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.404062 28661 raft_consensus.cc:2258] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:28.410024 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 76918.3us
W20240503 14:12:28.410342 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 7549b2ca31b04247bf3a2202e7585817 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7549b2ca31b04247bf3a2202e7585817 already in progress: creating tablet
W20240503 14:12:28.410650 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 3dfd6d40a57e4bea973874ff12c0ea3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dfd6d40a57e4bea973874ff12c0ea3a already in progress: creating tablet
W20240503 14:12:28.410795 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5500da0e5658492b840dd29273ab265f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5500da0e5658492b840dd29273ab265f already in progress: creating tablet
W20240503 14:12:28.410918 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 5685a6b3ea6943f8a72d74de87ac2166 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5685a6b3ea6943f8a72d74de87ac2166 already in progress: creating tablet
W20240503 14:12:28.411041 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 91eb4433685b4119a170054a0a777e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91eb4433685b4119a170054a0a777e05 already in progress: creating tablet
W20240503 14:12:28.411160 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a because the tablet was not found. No further retry: Not found: Tablet not found: f0fd8fb8ea1a4fc3bfa4f55ea59ae12a
I20240503 14:12:28.411273 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2cad43f3c6094253ba87d0ed3fc29fd9 (table default.Table_a9e8bd96c6824e9c8c64036f0848c876 [id=9e438ac1c3254c7aae6f81c707f433d7]) successfully deleted
I20240503 14:12:28.411418 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 9ac2833a0bb045dcb2824468d936ffe4 (table default.Table_472814febc0548599d68e796d57f48a1 [id=7128b9ce63544947ab63aa6d2490c691]) successfully deleted
W20240503 14:12:28.411542 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d93fea5caec94695a5df26d3630c50c7 because the tablet was not found. No further retry: Not found: Tablet not found: d93fea5caec94695a5df26d3630c50c7
I20240503 14:12:28.411649 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 11e14e8b5f4046029af4673a277112d9 (table default.Table_68199a1d18244f54bb0b7b0256a58e2a [id=2528ba5e3fe74c40949dcc7c289d0630]) successfully deleted
W20240503 14:12:28.411773 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 11ac912a3bd64cdc9082596e09c72e13 because the tablet was not found. No further retry: Not found: Tablet not found: 11ac912a3bd64cdc9082596e09c72e13
I20240503 14:12:28.411882 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 02d45788ae644a8581acca54cc55dd3f (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
W20240503 14:12:28.411994 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet dcd405fd84d44076b1792b7fcbe00e84 because the tablet was not found. No further retry: Not found: Tablet not found: dcd405fd84d44076b1792b7fcbe00e84
W20240503 14:12:28.412148 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 4e8411f474c247c2abfd0bd9682fb8a2 because the tablet was not found. No further retry: Not found: Tablet not found: 4e8411f474c247c2abfd0bd9682fb8a2
I20240503 14:12:28.412384 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet b2bba6a3d6e84552b027f23a29cbb38a (table default.Table_253d4e5a90d94897874db169bc3718b6 [id=ae3e54d70f4541898b7f4d03b4128ac9]) successfully deleted
I20240503 14:12:28.413033 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 (table default.Table_2e88623765c84939b082bcf13612fd61 [id=7ca3beb0b91840d08d178367a46c9936]) successfully deleted
I20240503 14:12:28.413156 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet dcd405fd84d44076b1792b7fcbe00e84 (table default.Table_92abfaed91ab4683a6b2e501ce673e58 [id=6539aa29cb864a2fa19c8a2f1ac7bd19]) successfully deleted
I20240503 14:12:28.413268 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 07354861ee6142d6b1fbe573397d3160 (table default.Table_7fbc01371cd145eeb7b51110d14f6456 [id=2cc7a41775044f4a993bd36fc229d152]) successfully deleted
W20240503 14:12:28.413431 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6a27d21de6a6441d810f379bba88f9a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a27d21de6a6441d810f379bba88f9a2 already in progress: creating tablet
W20240503 14:12:28.413607 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5de9edfb0d064939bda2ec7a9e0e500b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9edfb0d064939bda2ec7a9e0e500b already in progress: creating tablet
W20240503 14:12:28.413959 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0d4c5648ddc84407a8d8b44f2d63dcee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d4c5648ddc84407a8d8b44f2d63dcee already in progress: creating tablet
W20240503 14:12:28.414124 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 19d43cc864064c4c907ec0629391da10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19d43cc864064c4c907ec0629391da10 already in progress: creating tablet
W20240503 14:12:28.414287 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 13f8553cfbab4af2a38a73e81fed2a5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13f8553cfbab4af2a38a73e81fed2a5f already in progress: creating tablet
W20240503 14:12:28.414415 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9f5b973ea16d42babd2cc75e32624854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f5b973ea16d42babd2cc75e32624854 already in progress: creating tablet
W20240503 14:12:28.414628 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet addc2a6195034b0a927fdee7a2955122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet addc2a6195034b0a927fdee7a2955122 already in progress: creating tablet
I20240503 14:12:28.414952 28600 tablet_bootstrap.cc:492] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
W20240503 14:12:28.414953 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 77511682d71a40f5933edcb6a6523649 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77511682d71a40f5933edcb6a6523649 already in progress: creating tablet
W20240503 14:12:28.415099 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c99c15b4d13647e98cae1375b7f13827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c99c15b4d13647e98cae1375b7f13827 already in progress: creating tablet
I20240503 14:12:28.415115 28600 ts_tablet_manager.cc:1367] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
W20240503 14:12:28.415222 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ed4dc732bcb44222b4b15e5d77defe33 because the tablet was not found. No further retry: Not found: Tablet not found: ed4dc732bcb44222b4b15e5d77defe33
I20240503 14:12:28.415338 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 11ac912a3bd64cdc9082596e09c72e13 (table default.Table_78f62674ace541fabf9047f21a7cadbc [id=aed2a71b61da4d3981c77404ba8dffaa]) successfully deleted
W20240503 14:12:28.415465 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 495190fca06a40a386c8215eb2195920 because the tablet was not found. No further retry: Not found: Tablet not found: 495190fca06a40a386c8215eb2195920
I20240503 14:12:28.415572 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 57ab077988364bd687e54f90c3c265d5 (table default.Table_cdaf5876f13c4bd3adb7cb8aea0cfc88 [id=7877e3c7f4b14fcebfd7bec2e15b6240]) successfully deleted
I20240503 14:12:28.415675 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet f0fd8fb8ea1a4fc3bfa4f55ea59ae12a (table default.Table_11703c3db99a47cc85741ba32923f695 [id=7448b8a18a6d45c48bc56d1379672494]) successfully deleted
W20240503 14:12:28.415799 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet dda39f0323ec47799d3d44c96f3ba605 because the tablet was not found. No further retry: Not found: Tablet not found: dda39f0323ec47799d3d44c96f3ba605
W20240503 14:12:28.415916 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e4a544fc27624dcc8a4cb003e59408f2 because the tablet was not found. No further retry: Not found: Tablet not found: e4a544fc27624dcc8a4cb003e59408f2
W20240503 14:12:28.416031 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7b7ca19dd15f48a68ace7fefc7ae660f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b7ca19dd15f48a68ace7fefc7ae660f already in progress: creating tablet
I20240503 14:12:28.416075 28600 raft_consensus.cc:348] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.416414 28600 raft_consensus.cc:374] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.416477 28600 raft_consensus.cc:729] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.416780 28600 consensus_queue.cc:260] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:28.416941 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7d348da459634561985c4ac1710c0bb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d348da459634561985c4ac1710c0bb2 already in progress: creating tablet
W20240503 14:12:28.417784 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d2eb8328e86a4ee5a53647a810729519 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2eb8328e86a4ee5a53647a810729519 already in progress: creating tablet
I20240503 14:12:28.418380 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]) successfully deleted
W20240503 14:12:28.418774 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet bab214b7ec58425d8cd1180ee16b273d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab214b7ec58425d8cd1180ee16b273d already in progress: creating tablet
W20240503 14:12:28.419030 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 728ba3ac1cb645168481b84b5b548140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728ba3ac1cb645168481b84b5b548140 already in progress: creating tablet
W20240503 14:12:28.419252 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c725628f72884086b523b56ecf876069 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c725628f72884086b523b56ecf876069 already in progress: creating tablet
W20240503 14:12:28.419479 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ae1cd28bc07e4ccd9097f6dee471c5c6 because the tablet was not found. No further retry: Not found: Tablet not found: ae1cd28bc07e4ccd9097f6dee471c5c6
W20240503 14:12:28.419721 28017 consensus_peers.cc:485] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238 -> Peer bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571): Couldn't send request to peer bb6a7aee99214df4b02858ddae025b14. Status: Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:28.420245 28600 ts_tablet_manager.cc:1398] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:28.420774 28600 tablet_bootstrap.cc:492] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.421586 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 4897a41149574defbd3eae075c51879a (DEFAULT_TABLE table=default.Table_26feeadb423f43bcaa8a213736a1e877 [id=7d1430092d8d470195faf10c3807d2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.422019 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4897a41149574defbd3eae075c51879a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.422744 28600 tablet_bootstrap.cc:654] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.423563 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 4897a41149574defbd3eae075c51879a (DEFAULT_TABLE table=default.Table_26feeadb423f43bcaa8a213736a1e877 [id=7d1430092d8d470195faf10c3807d2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.430033 29996 consensus_queue.cc:1035] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240503 14:12:28.430949 28462 tablet_service.cc:1467] Processing CreateTablet for tablet f8f92b3d33204cb98aef3caf528b85e8 (DEFAULT_TABLE table=default.Table_1f00f0c6b19348a0a19c3045224268ef [id=154d054a9ec24701824e18f91d6261b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.430948 28187 tablet_service.cc:1467] Processing CreateTablet for tablet f8f92b3d33204cb98aef3caf528b85e8 (DEFAULT_TABLE table=default.Table_1f00f0c6b19348a0a19c3045224268ef [id=154d054a9ec24701824e18f91d6261b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.431638 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f92b3d33204cb98aef3caf528b85e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.433815 28197 tablet_service.cc:1467] Processing CreateTablet for tablet a88cf5de223f4a6abc21691fea1c4da0 (DEFAULT_TABLE table=default.Table_aa95e49d837a43cf9ed5ce0f5858cc45 [id=c7e5a81bd91447a193f8b5e57ba3911a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.434243 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88cf5de223f4a6abc21691fea1c4da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.448334 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4897a41149574defbd3eae075c51879a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.450057 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccb01614dee43e48d7e997bcfc7f6a6 (DEFAULT_TABLE table=default.Table_d6bd120570ec4a8ab95342666e7d1d83 [id=84f355c001064d28a408a8012a24df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.450570 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccb01614dee43e48d7e997bcfc7f6a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:28.456370 28427 consensus_peers.cc:485] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.457353 28456 tablet_service.cc:1467] Processing CreateTablet for tablet a88cf5de223f4a6abc21691fea1c4da0 (DEFAULT_TABLE table=default.Table_aa95e49d837a43cf9ed5ce0f5858cc45 [id=c7e5a81bd91447a193f8b5e57ba3911a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.457808 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88cf5de223f4a6abc21691fea1c4da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.420660 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 4897a41149574defbd3eae075c51879a (DEFAULT_TABLE table=default.Table_26feeadb423f43bcaa8a213736a1e877 [id=7d1430092d8d470195faf10c3807d2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.460433 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccb01614dee43e48d7e997bcfc7f6a6 (DEFAULT_TABLE table=default.Table_d6bd120570ec4a8ab95342666e7d1d83 [id=84f355c001064d28a408a8012a24df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.460639 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4897a41149574defbd3eae075c51879a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.460789 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccb01614dee43e48d7e997bcfc7f6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.463312 28600 tablet_bootstrap.cc:492] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.463479 28600 ts_tablet_manager.cc:1367] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.003s
I20240503 14:12:28.430191 28321 tablet_service.cc:1467] Processing CreateTablet for tablet f8f92b3d33204cb98aef3caf528b85e8 (DEFAULT_TABLE table=default.Table_1f00f0c6b19348a0a19c3045224268ef [id=154d054a9ec24701824e18f91d6261b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.464426 28600 raft_consensus.cc:348] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.464630 28600 raft_consensus.cc:374] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.464685 28600 raft_consensus.cc:729] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.464783 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f92b3d33204cb98aef3caf528b85e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.464901 28600 consensus_queue.cc:260] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.466781 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f92b3d33204cb98aef3caf528b85e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.438843 28352 raft_consensus.cc:3037] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.448115 28326 tablet_service.cc:1467] Processing CreateTablet for tablet a88cf5de223f4a6abc21691fea1c4da0 (DEFAULT_TABLE table=default.Table_aa95e49d837a43cf9ed5ce0f5858cc45 [id=c7e5a81bd91447a193f8b5e57ba3911a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.468732 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88cf5de223f4a6abc21691fea1c4da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.449252 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccb01614dee43e48d7e997bcfc7f6a6 (DEFAULT_TABLE table=default.Table_d6bd120570ec4a8ab95342666e7d1d83 [id=84f355c001064d28a408a8012a24df78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.472441 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccb01614dee43e48d7e997bcfc7f6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.476513 28600 ts_tablet_manager.cc:1398] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:12:28.477164 28600 tablet_bootstrap.cc:492] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.484670 29996 consensus_queue.cc:1035] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.487632 28661 ts_tablet_manager.cc:1875] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.487470 28600 tablet_bootstrap.cc:654] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.408226 29828 ts_tablet_manager.cc:1875] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.502331 28614 tablet_bootstrap.cc:492] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.503800 30272 raft_consensus.cc:482] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.503958 30272 raft_consensus.cc:504] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.504686 30272 leader_election.cc:288] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.505481 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:28.506112 28150 leader_election.cc:341] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.506708 28600 tablet_bootstrap.cc:492] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.506863 28600 ts_tablet_manager.cc:1367] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.005s
I20240503 14:12:28.507747 28600 raft_consensus.cc:348] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.507956 28600 raft_consensus.cc:374] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.508003 28600 raft_consensus.cc:729] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.511621 28600 consensus_queue.cc:260] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.514393 28600 ts_tablet_manager.cc:1398] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 14:12:28.514737 28600 tablet_bootstrap.cc:492] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.521036 28600 tablet_bootstrap.cc:654] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.529517 28661 ts_tablet_manager.cc:1888] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:28.529667 28661 log.cc:1196] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0a50b3e199314bfc8465fdb7b2425d07
I20240503 14:12:28.529980 28661 ts_tablet_manager.cc:1909] T 0a50b3e199314bfc8465fdb7b2425d07 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.530989 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1035) took 1699 ms. Trace:
I20240503 14:12:28.531098 28661 rpcz_store.cc:276] 0503 14:12:26.831307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.831357 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:28.360835 (+1529478us) ts_tablet_manager.cc:1120] Deleting tablet 0a50b3e199314bfc8465fdb7b2425d07
0503 14:12:28.360848 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.498724 (+137876us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.529356 (+ 30632us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.530959 (+ 1603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529289}
I20240503 14:12:28.534117 28661 tablet_replica.cc:330] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.534375 28661 raft_consensus.cc:2229] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.534571 28661 raft_consensus.cc:2258] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.502628 30315 raft_consensus.cc:482] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.505478 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.540161 30315 raft_consensus.cc:504] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.540966 30315 leader_election.cc:288] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:28.541558 28150 leader_election.cc:341] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.541688 28150 leader_election.cc:302] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.542143 30272 raft_consensus.cc:2726] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.543781 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d1314841574fc8bcf3ccb16755e421" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:28.544098 28290 leader_election.cc:341] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a3d1314841574fc8bcf3ccb16755e421
I20240503 14:12:28.544063 28493 raft_consensus.cc:2379] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:28.545224 28614 ts_tablet_manager.cc:1367] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.145s user 0.002s sys 0.002s
I20240503 14:12:28.546332 28290 leader_election.cc:302] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.546667 30315 raft_consensus.cc:3037] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.547665 28600 tablet_bootstrap.cc:492] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
W20240503 14:12:28.548951 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet b24ab3a5bba04a5092225a981f292222 (table default.Table_5354b912229e43539928c0e0df20a641 [id=294277efc25b401291a90d0fa70c90a9], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet b24ab3a5bba04a5092225a981f292222 consensus state has no leader
I20240503 14:12:28.549132 28600 ts_tablet_manager.cc:1367] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.005s
I20240503 14:12:28.550184 30315 raft_consensus.cc:2726] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:28.550870 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 101307us
I20240503 14:12:28.551162 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 4fad4508a29d4822b81af6947b57e7ef (table default.Table_68a22cfee7f644fcba9b9c3e34e6205c [id=0a3978674b74406484d7b5e9880f6220]) successfully deleted
I20240503 14:12:28.551573 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 8fe3affd0d144b1fb0078e7690b9b36d (table default.Table_32b3856710f446139fa555cd42254eef [id=cf15c50bd96d4420b98a06df477003b6]) successfully deleted
I20240503 14:12:28.551928 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ff4d28d3c2ea472191ad21355248a2fc (table default.Table_7b9e454770964e4594e5d043b06c2e78 [id=d4a1074afcca45b287b8d4416878f316]) successfully deleted
I20240503 14:12:28.552898 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 3d137ef4cfc2476cbc70f1cc72ad89e6 (table default.Table_2e88623765c84939b082bcf13612fd61 [id=7ca3beb0b91840d08d178367a46c9936]) successfully deleted
I20240503 14:12:28.552923 28600 raft_consensus.cc:348] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.553166 28600 raft_consensus.cc:374] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.553228 28600 raft_consensus.cc:729] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.553426 30272 raft_consensus.cc:482] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.553583 30272 raft_consensus.cc:504] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.554347 30272 leader_election.cc:288] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.554589 28600 consensus_queue.cc:260] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.555243 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007a892ef8504b848a400c05740eaa0c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.555317 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007a892ef8504b848a400c05740eaa0c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:28.555891 28150 leader_election.cc:341] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:28.556454 28150 leader_election.cc:341] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.556577 28150 leader_election.cc:302] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.556733 28600 ts_tablet_manager.cc:1398] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:28.557588 30272 raft_consensus.cc:2726] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.558630 28600 tablet_bootstrap.cc:492] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.562770 28661 ts_tablet_manager.cc:1875] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.565054 28604 tablet_bootstrap.cc:492] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.565235 28604 ts_tablet_manager.cc:1367] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.203s user 0.032s sys 0.009s
I20240503 14:12:28.566140 28604 raft_consensus.cc:348] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.566371 28604 raft_consensus.cc:374] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.566439 28604 raft_consensus.cc:729] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.566660 28604 consensus_queue.cc:260] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.567662 28604 ts_tablet_manager.cc:1398] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:28.568219 28604 tablet_bootstrap.cc:492] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.568905 29996 raft_consensus.cc:482] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.569067 29996 raft_consensus.cc:504] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.569898 29996 leader_election.cc:288] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.570458 28604 tablet_bootstrap.cc:654] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.570860 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf810e84367f420b928a171790849445" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
W20240503 14:12:28.571727 28427 leader_election.cc:341] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.573022 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf810e84367f420b928a171790849445" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:28.574021 28427 leader_election.cc:341] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.574167 28427 leader_election.cc:302] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:28.574637 29996 raft_consensus.cc:2726] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.575364 28614 raft_consensus.cc:348] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.575673 28614 raft_consensus.cc:374] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.575731 28614 raft_consensus.cc:729] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.575981 28614 consensus_queue.cc:260] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.579057 28600 tablet_bootstrap.cc:654] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:28.582633 30384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:28.584038 30379 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0690a516c89446d85c88cdf64fa4238)
I20240503 14:12:28.584456 30379 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6a7aee99214df4b02858ddae025b14" member_type: VOTER last_known_addr { host: "127.27.50.254" port: 36571 } } peers { permanent_uuid: "a10ddca38c26462eb221d6026e88c6cc" member_type: VOTER last_known_addr { host: "127.27.50.253" port: 44259 } } peers { permanent_uuid: "c0690a516c89446d85c88cdf64fa4238" member_type: VOTER last_known_addr { host: "127.27.50.252" port: 45363 } }
W20240503 14:12:28.585551 30386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:28.585754 30379 leader_election.cc:288] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571), c0690a516c89446d85c88cdf64fa4238 (127.27.50.252:45363)
I20240503 14:12:28.590009 28661 ts_tablet_manager.cc:1888] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.590154 28661 log.cc:1196] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/d93fea5caec94695a5df26d3630c50c7
I20240503 14:12:28.590502 28661 ts_tablet_manager.cc:1909] T d93fea5caec94695a5df26d3630c50c7 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.591380 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1036) took 1759 ms. Trace:
I20240503 14:12:28.591490 28661 rpcz_store.cc:276] 0503 14:12:26.831482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.831552 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:28.534014 (+1702462us) ts_tablet_manager.cc:1120] Deleting tablet d93fea5caec94695a5df26d3630c50c7
0503 14:12:28.534026 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.574381 (+ 40355us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.589872 (+ 15491us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.591344 (+ 1472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702314}
I20240503 14:12:28.591882 28661 tablet_replica.cc:330] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.592475 28600 tablet_bootstrap.cc:492] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.592666 28600 ts_tablet_manager.cc:1367] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240503 14:12:28.593006 28604 tablet_bootstrap.cc:492] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.593163 28604 ts_tablet_manager.cc:1367] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 14:12:28.593570 28600 raft_consensus.cc:348] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.593791 28600 raft_consensus.cc:374] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.593864 28600 raft_consensus.cc:729] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.593983 28604 raft_consensus.cc:348] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.594214 28604 raft_consensus.cc:374] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.594270 28604 raft_consensus.cc:729] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.594538 28604 consensus_queue.cc:260] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.595168 28604 ts_tablet_manager.cc:1398] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:28.595868 28604 tablet_bootstrap.cc:492] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.597986 28604 tablet_bootstrap.cc:654] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.611483 28600 consensus_queue.cc:260] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:28.612242 28427 consensus_peers.cc:485] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11e14e8b5f4046029af4673a277112d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.612876 28661 raft_consensus.cc:2229] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.614774 28661 raft_consensus.cc:2258] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:28.581000 28024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 888c28447db64751adca55ae769d0a37 consensus state has no leader
W20240503 14:12:28.615509 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 62608.1us
I20240503 14:12:28.615670 28600 ts_tablet_manager.cc:1398] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20240503 14:12:28.615669 28024 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1207) took 1094 ms. Trace:
I20240503 14:12:28.582267 28030 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1208) took 1060 ms. Trace:
I20240503 14:12:28.588547 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a10ddca38c26462eb221d6026e88c6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1299 } } ignore_live_leader: false dest_uuid: "c0690a516c89446d85c88cdf64fa4238" is_pre_election: true
I20240503 14:12:28.615754 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet dcd405fd84d44076b1792b7fcbe00e84 (table default.Table_92abfaed91ab4683a6b2e501ce673e58 [id=6539aa29cb864a2fa19c8a2f1ac7bd19]) successfully deleted
I20240503 14:12:28.616351 28600 tablet_bootstrap.cc:492] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.616709 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 4084bd4fbf604042bb33ff03c3f8cc19 (table default.Table_3a0134977dde48a0849c29c36487c742 [id=119495cd0b64414690f0d19bcce2ed78]) successfully deleted
I20240503 14:12:28.615762 28024 rpcz_store.cc:276] 0503 14:12:27.521540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.751374 (+229834us) service_pool.cc:225] Handling call
0503 14:12:27.751815 (+ 441us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:27.752067 (+ 252us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:27.752328 (+ 261us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:27.752397 (+ 69us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:28.580715 (+828318us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:28.615639 (+ 34924us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.766414 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.766511 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.766517 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:27.766520 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:27.766622 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.766636 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.766640 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:27.766718 (+ 78us) tablet.cc:699] Row locks acquired
0503 14:12:27.766721 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:27.766764 (+ 43us) write_op.cc:270] Start()
0503 14:12:27.766799 (+ 35us) write_op.cc:276] Timestamp: P: 1714745547766733 usec, L: 0
0503 14:12:27.766803 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.766981 (+ 178us) log.cc:842] Serialized 405 byte log entry
0503 14:12:28.557783 (+790802us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.580158 (+ 22375us) write_op.cc:301] APPLY: starting
0503 14:12:28.580179 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.580224 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.580258 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.580329 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.580373 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:28.580394 (+ 21us) write_op.cc:312] APPLY: finished
0503 14:12:28.580494 (+ 100us) log.cc:842] Serialized 23 byte log entry
0503 14:12:28.580578 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.580601 (+ 23us) write_op.cc:454] Released schema lock
0503 14:12:28.580608 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.581167 (+ 559us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.581174 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":22253,"apply.run_cpu_time_us":616,"apply.run_wall_time_us":1084,"num_ops":1,"prepare.queue_time_us":13442,"prepare.run_cpu_time_us":641,"prepare.run_wall_time_us":642,"replication_time_us":790943,"thread_start_us":22136,"threads_started":1}]]}
W20240503 14:12:28.618558 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 8410e8b6b511403ea609b1353855937b because the tablet was not found. No further retry: Not found: Tablet not found: 8410e8b6b511403ea609b1353855937b
I20240503 14:12:28.618764 28600 tablet_bootstrap.cc:654] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.615809 28030 rpcz_store.cc:276] 0503 14:12:27.521606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.752556 (+230950us) service_pool.cc:225] Handling call
0503 14:12:27.753479 (+ 923us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:27.753967 (+ 488us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.581945 (+827978us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.582081 (+ 136us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.582195 (+ 114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.767085 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.767173 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.767179 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:27.767182 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:27.767258 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.767268 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.767273 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:27.767379 (+ 106us) tablet.cc:699] Row locks acquired
0503 14:12:27.767382 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:27.767414 (+ 32us) write_op.cc:270] Start()
0503 14:12:27.767442 (+ 28us) write_op.cc:276] Timestamp: P: 1714745547767386 usec, L: 0
0503 14:12:27.767446 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.767590 (+ 144us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.580074 (+812484us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.581286 (+ 1212us) write_op.cc:301] APPLY: starting
0503 14:12:28.581303 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.581347 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.581374 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.581506 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.581547 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.581556 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:28.581641 (+ 85us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.581724 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.581747 (+ 23us) write_op.cc:454] Released schema lock
0503 14:12:28.581755 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.581839 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.581843 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1142,"apply.run_cpu_time_us":612,"apply.run_wall_time_us":610,"num_ops":2,"prepare.queue_time_us":12651,"prepare.run_cpu_time_us":555,"prepare.run_wall_time_us":554,"replication_time_us":812597,"threads_started":1}]]}
I20240503 14:12:28.619679 28614 ts_tablet_manager.cc:1398] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.066s user 0.002s sys 0.000s
I20240503 14:12:28.620045 28614 tablet_bootstrap.cc:492] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.619141 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 8c6202333af943cbbc0d09553c1c193c (table default.Table_fe566740b43c42658f0a0ff2de59038a [id=e9c7ec14573b44b79e70dc70fc9226c2]) successfully deleted
I20240503 14:12:28.622166 28614 tablet_bootstrap.cc:654] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:28.622496 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet bf21f6234dd64cf8a32f1599fe50746e because the tablet was not found. No further retry: Not found: Tablet not found: bf21f6234dd64cf8a32f1599fe50746e
I20240503 14:12:28.622813 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet d43823da09bb49be8f5d1a1149a4b11d (table default.Table_e65026502109470c9382acbdf03874d6 [id=99f3436cf65347648070d088f8430e1a]) successfully deleted
I20240503 14:12:28.626688 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e220336186f648419b186feada6f0124
I20240503 14:12:28.627166 28032 catalog_manager.cc:6554] ReplaceTablet: tablet e220336186f648419b186feada6f0124 deleted and replaced by tablet a83532bf2852473a9c51ce7cc78b5d34
I20240503 14:12:28.627301 28032 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1210) took 1062 ms. Trace:
I20240503 14:12:28.627379 28032 rpcz_store.cc:276] 0503 14:12:27.564894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.758617 (+193723us) service_pool.cc:225] Handling call
0503 14:12:28.627281 (+868664us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.767675 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.767763 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.767768 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:27.767770 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:27.767847 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.767854 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.767858 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:27.767913 (+ 55us) tablet.cc:699] Row locks acquired
0503 14:12:27.767915 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:27.767941 (+ 26us) write_op.cc:270] Start()
0503 14:12:27.767966 (+ 25us) write_op.cc:276] Timestamp: P: 1714745547767916 usec, L: 0
0503 14:12:27.767969 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.768127 (+ 158us) log.cc:842] Serialized 628 byte log entry
0503 14:12:28.625393 (+857266us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.625562 (+ 169us) write_op.cc:301] APPLY: starting
0503 14:12:28.625583 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.625638 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.625671 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.625809 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.625893 (+ 84us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:28.625903 (+ 10us) write_op.cc:312] APPLY: finished
0503 14:12:28.626016 (+ 113us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.626106 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.626136 (+ 30us) write_op.cc:454] Released schema lock
0503 14:12:28.626145 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.626215 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.626219 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":742,"apply.run_wall_time_us":742,"num_ops":2,"prepare.queue_time_us":7949,"prepare.run_cpu_time_us":503,"prepare.run_wall_time_us":502,"replication_time_us":857393}]]}
I20240503 14:12:28.629264 29828 ts_tablet_manager.cc:1888] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.629405 29828 log.cc:1196] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/920be6d87c4440d0bf7453c86cfe0b5c
I20240503 14:12:28.629817 29828 ts_tablet_manager.cc:1909] T 920be6d87c4440d0bf7453c86cfe0b5c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:28.632489 28026 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1211) took 1067 ms. Trace:
I20240503 14:12:28.632613 28026 rpcz_store.cc:276] 0503 14:12:27.564995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.760449 (+195454us) service_pool.cc:225] Handling call
0503 14:12:27.761375 (+ 926us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:27.761845 (+ 470us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.632185 (+870340us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.632316 (+ 131us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.632455 (+ 139us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.768213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.768292 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.768297 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:27.768299 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:27.768362 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.768370 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.768374 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:27.768438 (+ 64us) tablet.cc:699] Row locks acquired
0503 14:12:27.768441 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:27.768468 (+ 27us) write_op.cc:270] Start()
0503 14:12:27.768492 (+ 24us) write_op.cc:276] Timestamp: P: 1714745547768441 usec, L: 0
0503 14:12:27.768496 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.768619 (+ 123us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.625478 (+856859us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.626342 (+ 864us) write_op.cc:301] APPLY: starting
0503 14:12:28.626355 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.626393 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.626418 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.626520 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.626550 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.626558 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:28.626631 (+ 73us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.626701 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.626719 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:28.626726 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.626780 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.626784 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":830,"apply.run_cpu_time_us":492,"apply.run_wall_time_us":491,"num_ops":2,"prepare.queue_time_us":5934,"prepare.run_cpu_time_us":453,"prepare.run_wall_time_us":451,"replication_time_us":856968}]]}
I20240503 14:12:28.633072 30103 server_base.cc:1034] running on GCE node
I20240503 14:12:28.633921 28031 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1212) took 1068 ms. Trace:
I20240503 14:12:28.634033 28031 rpcz_store.cc:276] 0503 14:12:27.565063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.762963 (+197900us) service_pool.cc:225] Handling call
0503 14:12:27.763827 (+ 864us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:27.764296 (+ 469us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.633624 (+869328us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.633756 (+ 132us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.633890 (+ 134us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.768711 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.768786 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.768791 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:27.768793 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:27.768854 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.768861 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.768866 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:27.768917 (+ 51us) tablet.cc:699] Row locks acquired
0503 14:12:27.768919 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:27.768943 (+ 24us) write_op.cc:270] Start()
0503 14:12:27.768965 (+ 22us) write_op.cc:276] Timestamp: P: 1714745547768918 usec, L: 0
0503 14:12:27.768968 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.769085 (+ 117us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.625531 (+856446us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.626867 (+ 1336us) write_op.cc:301] APPLY: starting
0503 14:12:28.626880 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.626915 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.626940 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.627057 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.627090 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.627099 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:28.627161 (+ 62us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.627207 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.627229 (+ 22us) write_op.cc:454] Released schema lock
0503 14:12:28.627235 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.627298 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.627302 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1287,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":495,"num_ops":2,"prepare.queue_time_us":3934,"prepare.run_cpu_time_us":418,"prepare.run_wall_time_us":417,"replication_time_us":856550,"threads_started":1}]]}
I20240503 14:12:28.647641 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0a50b3e199314bfc8465fdb7b2425d07 (table default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0 [id=82ad92c83a0c44bc86486fb6137a47e3]) successfully deleted
W20240503 14:12:28.648540 30390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:28.650133 30103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:28.651106 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 978) took 2634 ms. Trace:
I20240503 14:12:28.651219 29828 rpcz_store.cc:276] 0503 14:12:26.016990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.017028 (+ 38us) service_pool.cc:225] Handling call
0503 14:12:28.406704 (+2389676us) ts_tablet_manager.cc:1120] Deleting tablet 920be6d87c4440d0bf7453c86cfe0b5c
0503 14:12:28.406715 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.501219 (+ 94504us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.629106 (+127887us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.651071 (+ 21965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2389532}
I20240503 14:12:28.651568 29828 tablet_replica.cc:330] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:28.651767 29828 raft_consensus.cc:2229] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.651978 29828 raft_consensus.cc:2258] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.652691 29828 ts_tablet_manager.cc:1875] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:28.653057 29436 leader_election.cc:334] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb6a7aee99214df4b02858ddae025b14 (127.27.50.254:36571): Network error: Client connection negotiation failed: client connection to 127.27.50.254:36571: connect: Connection refused (error 111)
I20240503 14:12:28.653431 29436 leader_election.cc:302] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a10ddca38c26462eb221d6026e88c6cc; no voters: bb6a7aee99214df4b02858ddae025b14, c0690a516c89446d85c88cdf64fa4238
I20240503 14:12:28.654474 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet d93fea5caec94695a5df26d3630c50c7 (table default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]) successfully deleted
I20240503 14:12:28.654973 30379 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a10ddca38c26462eb221d6026e88c6cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:28.655040 28661 ts_tablet_manager.cc:1875] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.655550 28600 tablet_bootstrap.cc:492] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
W20240503 14:12:28.657274 28025 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f223cbbcaa5402a888b084df0fe1c27 consensus state has no leader
I20240503 14:12:28.657454 28025 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1213) took 1092 ms. Trace:
I20240503 14:12:28.657517 28025 rpcz_store.cc:276] 0503 14:12:27.565128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.765092 (+199964us) service_pool.cc:225] Handling call
0503 14:12:27.765454 (+ 362us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:27.765690 (+ 236us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:27.765865 (+ 175us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:27.765941 (+ 76us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:28.657047 (+891106us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:28.657437 (+ 390us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.769156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.769229 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.769234 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:27.769236 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:27.769303 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.769311 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.769315 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:27.769357 (+ 42us) tablet.cc:699] Row locks acquired
0503 14:12:27.769359 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:27.769385 (+ 26us) write_op.cc:270] Start()
0503 14:12:27.769408 (+ 23us) write_op.cc:276] Timestamp: P: 1714745547769359 usec, L: 0
0503 14:12:27.769412 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.769543 (+ 131us) log.cc:842] Serialized 405 byte log entry
0503 14:12:28.656133 (+886590us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.656398 (+ 265us) write_op.cc:301] APPLY: starting
0503 14:12:28.656418 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.656495 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.656531 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.656595 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.656638 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:28.656648 (+ 10us) write_op.cc:312] APPLY: finished
0503 14:12:28.656753 (+ 105us) log.cc:842] Serialized 23 byte log entry
0503 14:12:28.656835 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.656858 (+ 23us) write_op.cc:454] Released schema lock
0503 14:12:28.656866 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.656929 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.656933 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":153,"apply.run_cpu_time_us":576,"apply.run_wall_time_us":589,"num_ops":1,"prepare.queue_time_us":2807,"prepare.run_cpu_time_us":433,"prepare.run_wall_time_us":431,"replication_time_us":886689}]]}
I20240503 14:12:28.658607 28604 tablet_bootstrap.cc:492] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.658779 28604 ts_tablet_manager.cc:1367] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.018s
I20240503 14:12:28.658775 28027 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1214) took 1093 ms. Trace:
I20240503 14:12:28.658869 28027 rpcz_store.cc:276] 0503 14:12:27.565204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.773032 (+207828us) service_pool.cc:225] Handling call
0503 14:12:27.773967 (+ 935us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:27.774434 (+ 467us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.658507 (+884073us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.658635 (+ 128us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.658752 (+ 117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.796364 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.796478 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.796484 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:27.796487 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:27.796577 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.796587 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.796592 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:27.796666 (+ 74us) tablet.cc:699] Row locks acquired
0503 14:12:27.796668 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:27.796724 (+ 56us) write_op.cc:270] Start()
0503 14:12:27.796761 (+ 37us) write_op.cc:276] Timestamp: P: 1714745547796688 usec, L: 0
0503 14:12:27.796765 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.797071 (+ 306us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.656223 (+859152us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.657023 (+ 800us) write_op.cc:301] APPLY: starting
0503 14:12:28.657035 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.657074 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.657102 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.657225 (+ 123us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.657260 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.657268 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:28.657344 (+ 76us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.657405 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.657422 (+ 17us) write_op.cc:454] Released schema lock
0503 14:12:28.657428 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.657476 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.657480 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":767,"apply.run_cpu_time_us":503,"apply.run_wall_time_us":502,"num_ops":2,"prepare.queue_time_us":21397,"prepare.run_cpu_time_us":751,"prepare.run_wall_time_us":792,"replication_time_us":859442}]]}
I20240503 14:12:28.659605 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_33246ca4181441eaa0e703cfd07c1b3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:28.660297 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33246ca4181441eaa0e703cfd07c1b3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:28.661131 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_47f02eacca0d41e3a4d9913ed4768151" } new_table_name: "default.Table_2fe7f4a5bc6f467e890e502b73d2545c" modify_external_catalogs: true
I20240503 14:12:28.661393 29994 raft_consensus.cc:482] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.661547 29994 raft_consensus.cc:504] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.661551 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_70505af08e3d4b4aa9b097f33cae154a" } new_table_name: "default.Table_9962abfb7b5a444585fcf4f86e9e74dc" modify_external_catalogs: true
I20240503 14:12:28.661970 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3d0d6e04014b414aa95434036f1b4516" } modify_external_catalogs: true
I20240503 14:12:28.662089 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ccd689a56b554370857c1833876a7851"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:28.662240 29994 leader_election.cc:288] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:28.662685 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccd689a56b554370857c1833876a7851 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:28.663264 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_8a699c7b01a04e4aad97227bb13be53a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:28.663820 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a699c7b01a04e4aad97227bb13be53a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:28.667966 28028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1209) took 1137 ms. Trace:
I20240503 14:12:28.668133 28600 ts_tablet_manager.cc:1367] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20240503 14:12:28.668155 28028 rpcz_store.cc:276] 0503 14:12:27.530712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.757405 (+226693us) service_pool.cc:225] Handling call
0503 14:12:27.777103 (+ 19698us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:27.777561 (+ 458us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.660476 (+882915us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.667806 (+ 7330us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.667936 (+ 130us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.797188 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.797719 (+ 531us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.797724 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:27.797727 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:27.797815 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.797825 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.797830 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:27.797896 (+ 66us) tablet.cc:699] Row locks acquired
0503 14:12:27.797899 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:27.797934 (+ 35us) write_op.cc:270] Start()
0503 14:12:27.797963 (+ 29us) write_op.cc:276] Timestamp: P: 1714745547797903 usec, L: 0
0503 14:12:27.797967 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.798211 (+ 244us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.656264 (+858053us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.657558 (+ 1294us) write_op.cc:301] APPLY: starting
0503 14:12:28.657569 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.657599 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.657624 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.657702 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.657736 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.657742 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:28.657798 (+ 56us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.657851 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.657866 (+ 15us) write_op.cc:454] Released schema lock
0503 14:12:28.657873 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.657928 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.657931 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1264,"apply.run_cpu_time_us":415,"apply.run_wall_time_us":415,"num_ops":2,"prepare.queue_time_us":19160,"prepare.run_cpu_time_us":643,"prepare.run_wall_time_us":1086,"replication_time_us":858283}]]}
I20240503 14:12:28.669049 28600 raft_consensus.cc:348] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.669281 28600 raft_consensus.cc:374] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.669080 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a7d0ab9ea1144a03899dd81992e69db2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:28.669348 28600 raft_consensus.cc:729] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.669587 28600 consensus_queue.cc:260] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:28.669699 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7d0ab9ea1144a03899dd81992e69db2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:28.671435 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3d0d6e04014b414aa95434036f1b4516" } modify_external_catalogs: true
I20240503 14:12:28.675736 28604 raft_consensus.cc:348] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.676021 28604 raft_consensus.cc:374] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:12:28.676849 28427 leader_election.cc:341] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2cad43f3c6094253ba87d0ed3fc29fd9
I20240503 14:12:28.678586 28604 raft_consensus.cc:729] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.678746 28600 ts_tablet_manager.cc:1398] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:12:28.679327 28604 consensus_queue.cc:260] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.680347 28604 ts_tablet_manager.cc:1398] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20240503 14:12:28.682734 29994 raft_consensus.cc:482] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.682883 29994 raft_consensus.cc:504] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.683063 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 920be6d87c4440d0bf7453c86cfe0b5c (table default.Table_d8cb0703edfb4e96ba081bd6f299d867 [id=6119a6b076964647bd7247e0a896a030]) successfully deleted
I20240503 14:12:28.683441 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet e220336186f648419b186feada6f0124 with delete_type TABLET_DATA_DELETED (replaced by tablet a83532bf2852473a9c51ce7cc78b5d34) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:28.683563 29994 leader_election.cc:288] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.683693 28604 tablet_bootstrap.cc:492] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.685809 28604 tablet_bootstrap.cc:654] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.676038 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad43f3c6094253ba87d0ed3fc29fd9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.686074 28352 raft_consensus.cc:2454] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.684917 30315 raft_consensus.cc:482] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.685415 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34238c9859724d97a32f2a1d9315edb1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.685844 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet e220336186f648419b186feada6f0124 with delete_type TABLET_DATA_DELETED (replaced by tablet a83532bf2852473a9c51ce7cc78b5d34) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:28.686381 30315 raft_consensus.cc:504] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.686925 30103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:28.687089 30315 leader_election.cc:288] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.689316 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e4f6e3908e420e9fa156ee24b437d6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.689570 28493 raft_consensus.cc:2454] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:28.690171 28290 leader_election.cc:302] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.690562 30315 raft_consensus.cc:2781] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.690647 30315 raft_consensus.cc:482] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.690709 30315 raft_consensus.cc:3037] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.693362 30315 raft_consensus.cc:504] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.694080 30315 leader_election.cc:288] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:28.694689 28290 leader_election.cc:341] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:28.695372 30103 hybrid_clock.cc:648] HybridClock initialized: now 1714745548695294 us; error 67 us; skew 500 ppm
I20240503 14:12:28.695636 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e4f6e3908e420e9fa156ee24b437d6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:28.695838 28493 raft_consensus.cc:3037] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.695876 30103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:28.698549 28604 tablet_bootstrap.cc:492] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
W20240503 14:12:28.698772 28290 leader_election.cc:341] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:28.698824 28604 ts_tablet_manager.cc:1367] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 14:12:28.698549 28493 raft_consensus.cc:2454] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:28.699483 28290 leader_election.cc:302] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:28.699959 30315 raft_consensus.cc:2781] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.699970 28604 raft_consensus.cc:348] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.700718 30315 raft_consensus.cc:686] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:28.700749 28604 raft_consensus.cc:374] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.701035 28604 raft_consensus.cc:729] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.701031 30315 consensus_queue.cc:237] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.701551 28604 consensus_queue.cc:260] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.702633 28604 ts_tablet_manager.cc:1398] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:28.703722 28604 tablet_bootstrap.cc:492] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.704726 28661 ts_tablet_manager.cc:1888] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.704876 28661 log.cc:1196] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/20b176a071ba4f3fbff2a64d6e46027a
I20240503 14:12:28.705266 28661 ts_tablet_manager.cc:1909] T 20b176a071ba4f3fbff2a64d6e46027a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.705523 28427 leader_election.cc:302] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
W20240503 14:12:28.706070 28427 leader_election.cc:341] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.706512 28614 tablet_bootstrap.cc:492] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.706691 28614 ts_tablet_manager.cc:1367] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20240503 14:12:28.707859 28604 tablet_bootstrap.cc:654] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.679111 28600 tablet_bootstrap.cc:492] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.683992 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34238c9859724d97a32f2a1d9315edb1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.709951 28212 raft_consensus.cc:2454] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.683737 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet e220336186f648419b186feada6f0124 with delete_type TABLET_DATA_DELETED (replaced by tablet a83532bf2852473a9c51ce7cc78b5d34) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.688573 30272 consensus_queue.cc:1035] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20240503 14:12:28.710887 28427 leader_election.cc:302] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:28.711434 30253 raft_consensus.cc:2781] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.711542 30253 raft_consensus.cc:482] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.711607 30253 raft_consensus.cc:3037] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.714707 30253 raft_consensus.cc:504] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.714901 29996 raft_consensus.cc:2781] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.715000 29996 raft_consensus.cc:482] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.715061 29996 raft_consensus.cc:3037] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.715481 30253 leader_election.cc:288] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.718008 29996 raft_consensus.cc:504] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.718719 29996 leader_election.cc:288] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.721967 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cad43f3c6094253ba87d0ed3fc29fd9" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:28.722209 28355 raft_consensus.cc:3037] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.707504 28614 raft_consensus.cc:348] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.722379 28614 raft_consensus.cc:374] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.722595 28614 raft_consensus.cc:729] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.722873 28614 consensus_queue.cc:260] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.723515 28614 ts_tablet_manager.cc:1398] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20240503 14:12:28.723685 30103 webserver.cc:460] Webserver started at http://127.27.50.254:38435/ using document root <none> and password file <none>
I20240503 14:12:28.723843 28614 tablet_bootstrap.cc:492] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.725023 30103 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:28.725359 30103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:28.725627 28355 raft_consensus.cc:2454] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:28.725970 28614 tablet_bootstrap.cc:654] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.733876 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34238c9859724d97a32f2a1d9315edb1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:28.733937 30266 consensus_queue.cc:1035] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.735555 28427 leader_election.cc:302] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.736379 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34238c9859724d97a32f2a1d9315edb1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:28.736596 28212 raft_consensus.cc:3037] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.737332 28030 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1215) took 1171 ms. Trace:
I20240503 14:12:28.737447 30272 raft_consensus.cc:482] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.737586 30272 raft_consensus.cc:504] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.737437 28030 rpcz_store.cc:276] 0503 14:12:27.565698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.619160 (+1053462us) service_pool.cc:225] Handling call
0503 14:12:28.662002 (+ 42842us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:28.664253 (+ 2251us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:28.664761 (+ 508us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.737058 (+ 72297us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.737193 (+ 135us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.737307 (+ 114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.677121 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.677278 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.677284 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.677287 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:28.677415 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.677430 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.677435 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.677527 (+ 92us) tablet.cc:699] Row locks acquired
0503 14:12:28.677530 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.677580 (+ 50us) write_op.cc:270] Start()
0503 14:12:28.677619 (+ 39us) write_op.cc:276] Timestamp: P: 1714745548677560 usec, L: 0
0503 14:12:28.677624 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.677888 (+ 264us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.736268 (+ 58380us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.736449 (+ 181us) write_op.cc:301] APPLY: starting
0503 14:12:28.736470 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.736530 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.736563 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.736704 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.736747 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.736756 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:28.736853 (+ 97us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.736930 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.736956 (+ 26us) write_op.cc:454] Released schema lock
0503 14:12:28.736964 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.737031 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.737034 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":97815168,"child_traces":[["op",{"apply.queue_time_us":117,"apply.run_cpu_time_us":639,"apply.run_wall_time_us":638,"num_ops":2,"prepare.queue_time_us":11810,"prepare.run_cpu_time_us":955,"prepare.run_wall_time_us":3854,"raft.queue_time_us":2792,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":58609,"thread_start_us":13091,"threads_started":1}]]}
I20240503 14:12:28.738150 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f642b306a8c49fd9c070a9d340889a2 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:28.738238 30272 leader_election.cc:288] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:28.738617 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f53643c4a69648989a4bd0c30c5dd24f has no consensus state
I20240503 14:12:28.738765 28032 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1216) took 1163 ms. Trace:
I20240503 14:12:28.738857 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7779f7a9544e17aea7135eab2ccd5f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.738844 28032 rpcz_store.cc:276] 0503 14:12:27.575126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.628003 (+1052877us) service_pool.cc:225] Handling call
0503 14:12:28.661091 (+ 33088us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cee0f46 00007fe123ed3c26 00007fe123ed3955 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:28.665499 (+ 4408us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.665747 (+ 248us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:28.665931 (+ 184us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:28.666017 (+ 86us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:28.738406 (+ 72389us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:28.738748 (+ 342us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.681026 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.681142 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.681148 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.681151 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:28.681255 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.681267 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.681273 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:28.681335 (+ 62us) tablet.cc:699] Row locks acquired
0503 14:12:28.681338 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.681382 (+ 44us) write_op.cc:270] Start()
0503 14:12:28.681418 (+ 36us) write_op.cc:276] Timestamp: P: 1714745548681363 usec, L: 0
0503 14:12:28.681423 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.681631 (+ 208us) log.cc:842] Serialized 405 byte log entry
0503 14:12:28.736355 (+ 54724us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.737125 (+ 770us) write_op.cc:301] APPLY: starting
0503 14:12:28.737135 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.737170 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.737193 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.737235 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.737263 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:28.737269 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:28.737341 (+ 72us) log.cc:842] Serialized 23 byte log entry
0503 14:12:28.737411 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.737427 (+ 16us) write_op.cc:454] Released schema lock
0503 14:12:28.737433 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.737483 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.737487 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":75428736,"child_traces":[["op",{"apply.queue_time_us":739,"apply.run_cpu_time_us":409,"apply.run_wall_time_us":408,"num_ops":1,"prepare.queue_time_us":14526,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":3200,"raft.queue_time_us":327,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":54916}]]}
I20240503 14:12:28.739492 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet f67c4cb980b44e9397bea70363fe1acd from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:28.739667 28150 leader_election.cc:341] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.740053 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet a3d1314841574fc8bcf3ccb16755e421 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:28.740541 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cdaf5876f13c4bd3adb7cb8aea0cfc88" } modify_external_catalogs: true
I20240503 14:12:28.740708 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cdaf5876f13c4bd3adb7cb8aea0cfc88" } modify_external_catalogs: true
I20240503 14:12:28.741550 28600 tablet_bootstrap.cc:654] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.738889 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7779f7a9544e17aea7135eab2ccd5f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.741905 28212 raft_consensus.cc:2454] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:28.742745 29994 raft_consensus.cc:2781] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.742879 29994 raft_consensus.cc:686] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
W20240503 14:12:28.742998 28150 leader_election.cc:341] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.743137 28150 leader_election.cc:302] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.743119 29994 consensus_queue.cc:237] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.743501 30272 raft_consensus.cc:2726] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.750643 28604 tablet_bootstrap.cc:492] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.750840 28604 ts_tablet_manager.cc:1367] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.008s
I20240503 14:12:28.751745 28604 raft_consensus.cc:348] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.751974 28604 raft_consensus.cc:374] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.752027 28604 raft_consensus.cc:729] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.752306 28604 consensus_queue.cc:260] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.752966 28604 ts_tablet_manager.cc:1398] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:28.753305 28604 tablet_bootstrap.cc:492] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:28.754169 28427 leader_election.cc:341] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:28.754673 28427 leader_election.cc:341] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 2cad43f3c6094253ba87d0ed3fc29fd9
I20240503 14:12:28.755278 28604 tablet_bootstrap.cc:654] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.756024 29828 ts_tablet_manager.cc:1888] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.763535 29828 log.cc:1196] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/b2bba6a3d6e84552b027f23a29cbb38a
I20240503 14:12:28.764133 28604 tablet_bootstrap.cc:492] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.764184 29828 ts_tablet_manager.cc:1909] T b2bba6a3d6e84552b027f23a29cbb38a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:28.764305 28604 ts_tablet_manager.cc:1367] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20240503 14:12:28.765159 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 979) took 2747 ms. Trace:
I20240503 14:12:28.766006 29828 rpcz_store.cc:276] 0503 14:12:26.017275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.017312 (+ 37us) service_pool.cc:225] Handling call
0503 14:12:28.651517 (+2634205us) ts_tablet_manager.cc:1120] Deleting tablet b2bba6a3d6e84552b027f23a29cbb38a
0503 14:12:28.651526 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.700887 (+ 49361us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.755895 (+ 55008us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.765133 (+ 9238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2634059}
I20240503 14:12:28.766162 28027 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1220) took 1108 ms. Trace:
I20240503 14:12:28.766652 29828 tablet_replica.cc:330] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:28.766254 28027 rpcz_store.cc:276] 0503 14:12:27.657515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.659389 (+1001874us) service_pool.cc:225] Handling call
0503 14:12:28.664154 (+ 4765us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:28.664758 (+ 604us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.765881 (+101123us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.766018 (+ 137us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.766130 (+ 112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.684255 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.684376 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.684383 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:28.684385 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.687369 (+ 2984us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.687382 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.687388 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.687478 (+ 90us) tablet.cc:699] Row locks acquired
0503 14:12:28.687481 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.687534 (+ 53us) write_op.cc:270] Start()
0503 14:12:28.687575 (+ 41us) write_op.cc:276] Timestamp: P: 1714745548687514 usec, L: 0
0503 14:12:28.687579 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.687829 (+ 250us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.764765 (+ 76936us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.765230 (+ 465us) write_op.cc:301] APPLY: starting
0503 14:12:28.765251 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.765313 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.765364 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.765495 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.765537 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.765545 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:28.765667 (+ 122us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.765754 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.765779 (+ 25us) write_op.cc:454] Released schema lock
0503 14:12:28.765787 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.765859 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.765863 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":23680,"child_traces":[["op",{"apply.queue_time_us":391,"apply.run_cpu_time_us":697,"apply.run_wall_time_us":697,"mutex_wait_us":1234,"num_ops":2,"prepare.queue_time_us":18962,"prepare.run_cpu_time_us":914,"prepare.run_wall_time_us":4772,"raft.queue_time_us":498,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":77155}]]}
I20240503 14:12:28.767084 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet b2bba6a3d6e84552b027f23a29cbb38a (table default.Table_253d4e5a90d94897874db169bc3718b6 [id=ae3e54d70f4541898b7f4d03b4128ac9]) successfully deleted
I20240503 14:12:28.767886 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1038) took 1935 ms. Trace:
I20240503 14:12:28.769444 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1de6580ff46348ffa25a2704e01ea777" } modify_external_catalogs: true
I20240503 14:12:28.769641 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1de6580ff46348ffa25a2704e01ea777" } modify_external_catalogs: true
W20240503 14:12:28.769723 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet beba0d09f5d34980ba7ed463ff9b4c34 consensus state has no leader
I20240503 14:12:28.769884 28031 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1218) took 1115 ms. Trace:
I20240503 14:12:28.770210 28661 rpcz_store.cc:276] 0503 14:12:26.832062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.832133 (+ 71us) service_pool.cc:225] Handling call
0503 14:12:28.591811 (+1759678us) ts_tablet_manager.cc:1120] Deleting tablet 20b176a071ba4f3fbff2a64d6e46027a
0503 14:12:28.591822 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.680987 (+ 89165us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.704600 (+ 23613us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.767858 (+ 63258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61656,"tablet-delete.queue_time_us":1759495}
I20240503 14:12:28.769974 28031 rpcz_store.cc:276] 0503 14:12:27.654521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.634621 (+980100us) service_pool.cc:225] Handling call
0503 14:12:28.661513 (+ 26892us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cee0f46 00007fe123ed3c26 00007fe123ed3955 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:28.671970 (+ 10457us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.672174 (+ 204us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:28.672390 (+ 216us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:28.672472 (+ 82us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:28.769494 (+ 97022us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:28.769864 (+ 370us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.690788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.690902 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.690908 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.690910 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.691014 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.691025 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.691030 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:28.691093 (+ 63us) tablet.cc:699] Row locks acquired
0503 14:12:28.691096 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.691135 (+ 39us) write_op.cc:270] Start()
0503 14:12:28.691169 (+ 34us) write_op.cc:276] Timestamp: P: 1714745548691117 usec, L: 0
0503 14:12:28.691172 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.691358 (+ 186us) log.cc:842] Serialized 405 byte log entry
0503 14:12:28.764882 (+ 73524us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.768017 (+ 3135us) write_op.cc:301] APPLY: starting
0503 14:12:28.768035 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.768077 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.768126 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.768180 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.768213 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:28.768220 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:28.768316 (+ 96us) log.cc:842] Serialized 23 byte log entry
0503 14:12:28.768383 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.768404 (+ 21us) write_op.cc:454] Released schema lock
0503 14:12:28.768412 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.768475 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.768478 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":61192448,"child_traces":[["op",{"apply.queue_time_us":3078,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":524,"num_ops":1,"prepare.queue_time_us":17751,"prepare.run_cpu_time_us":717,"prepare.run_wall_time_us":1561,"raft.queue_time_us":44,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":73697}]]}
I20240503 14:12:28.770805 28661 tablet_replica.cc:330] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.770813 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e4155be94a44f6c8a8683824f24925a" } modify_external_catalogs: true
I20240503 14:12:28.770978 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2e4155be94a44f6c8a8683824f24925a" } modify_external_catalogs: true
I20240503 14:12:28.771024 28661 raft_consensus.cc:2229] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.771451 28661 raft_consensus.cc:2258] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.771664 28025 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1219) took 1114 ms. Trace:
I20240503 14:12:28.771986 30272 raft_consensus.cc:482] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.765334 28604 raft_consensus.cc:348] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.772172 30272 raft_consensus.cc:504] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.772282 28604 raft_consensus.cc:374] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.772344 28604 raft_consensus.cc:729] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.771756 28025 rpcz_store.cc:276] 0503 14:12:27.657421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.658233 (+1000812us) service_pool.cc:225] Handling call
0503 14:12:28.663159 (+ 4926us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedecb3 00007fe123ed02a6 00007fe123ecffd5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:28.674369 (+ 11210us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:28.674897 (+ 528us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.771419 (+ 96522us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.771539 (+ 120us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.771642 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.692365 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.692476 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.692482 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.692485 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:28.692573 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.692584 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.692589 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.692663 (+ 74us) tablet.cc:699] Row locks acquired
0503 14:12:28.692666 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.692700 (+ 34us) write_op.cc:270] Start()
0503 14:12:28.692732 (+ 32us) write_op.cc:276] Timestamp: P: 1714745548692683 usec, L: 0
0503 14:12:28.692736 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.692924 (+ 188us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.764922 (+ 71998us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.768563 (+ 3641us) write_op.cc:301] APPLY: starting
0503 14:12:28.768575 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.768615 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.768641 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.768750 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.768794 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.768804 (+ 10us) write_op.cc:312] APPLY: finished
0503 14:12:28.768873 (+ 69us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.768920 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.768936 (+ 16us) write_op.cc:454] Released schema lock
0503 14:12:28.768942 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.769001 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.769005 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":10606464,"child_traces":[["op",{"apply.queue_time_us":3611,"apply.run_cpu_time_us":489,"apply.run_wall_time_us":489,"num_ops":2,"prepare.queue_time_us":16983,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":1343,"raft.queue_time_us":40,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":72172,"thread_start_us":1061,"threads_started":1}]]}
I20240503 14:12:28.772536 28604 consensus_queue.cc:260] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.772727 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f3380746b00d4e5f9ba9f5851075ec37" } modify_external_catalogs: true
I20240503 14:12:28.772874 30272 leader_election.cc:288] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.772890 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f3380746b00d4e5f9ba9f5851075ec37" } modify_external_catalogs: true
I20240503 14:12:28.773546 28028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1221) took 1113 ms. Trace:
I20240503 14:12:28.773587 29996 raft_consensus.cc:482] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.773672 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee9b69dd1f47219e7d3126207810c7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.773707 29996 raft_consensus.cc:504] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.773651 28028 rpcz_store.cc:276] 0503 14:12:27.660450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.668831 (+1008381us) service_pool.cc:225] Handling call
0503 14:12:28.669818 (+ 987us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:28.670310 (+ 492us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:28.773287 (+102977us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:28.773426 (+ 139us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:28.773524 (+ 98us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.689061 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.689174 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.689180 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.689181 (+ 1us) tablet.cc:660] Decoding operations
0503 14:12:28.689273 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.689284 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.689289 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.689369 (+ 80us) tablet.cc:699] Row locks acquired
0503 14:12:28.689372 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.689418 (+ 46us) write_op.cc:270] Start()
0503 14:12:28.689454 (+ 36us) write_op.cc:276] Timestamp: P: 1714745548689398 usec, L: 0
0503 14:12:28.689459 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.689677 (+ 218us) log.cc:842] Serialized 433 byte log entry
0503 14:12:28.764839 (+ 75162us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.769128 (+ 4289us) write_op.cc:301] APPLY: starting
0503 14:12:28.769141 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.769192 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.769217 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.769329 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.769365 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:28.769420 (+ 55us) write_op.cc:312] APPLY: finished
0503 14:12:28.769483 (+ 63us) log.cc:842] Serialized 29 byte log entry
0503 14:12:28.769530 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.769544 (+ 14us) write_op.cc:454] Released schema lock
0503 14:12:28.769550 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.769601 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.769604 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3060,"apply.run_cpu_time_us":494,"apply.run_wall_time_us":521,"num_ops":2,"prepare.queue_time_us":18229,"prepare.run_cpu_time_us":759,"prepare.run_wall_time_us":1688,"raft.queue_time_us":46,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":75363}]]}
I20240503 14:12:28.774284 29996 leader_election.cc:288] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:28.774240 28150 leader_election.cc:341] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.774459 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_00f5a3c1cea944eb91d3b4655518ae6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:28.774711 28604 ts_tablet_manager.cc:1398] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:12:28.775004 28604 tablet_bootstrap.cc:492] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:28.775059 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00f5a3c1cea944eb91d3b4655518ae6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:28.775470 28427 leader_election.cc:302] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:28.775718 29994 raft_consensus.cc:2781] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.775733 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.775835 29994 raft_consensus.cc:686] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:28.775954 28212 raft_consensus.cc:2454] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.776104 29994 consensus_queue.cc:237] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.779289 29996 consensus_queue.cc:1035] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:28.779305 28604 tablet_bootstrap.cc:654] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.783874 28661 ts_tablet_manager.cc:1875] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.786509 28427 leader_election.cc:302] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.788105 29994 raft_consensus.cc:2781] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.788290 29994 raft_consensus.cc:482] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.788302 30103 fs_manager.cc:709] Time spent opening directory manager: real 0.060s user 0.014s sys 0.005s
I20240503 14:12:28.788424 29994 raft_consensus.cc:3037] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.788817 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2]). Expected version 3 got 0
I20240503 14:12:28.789031 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:28.789384 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 77c77a11ef384709aef5b62273380573 (table default.Table_ece674e6e74547b6a4d04f1f2b167a08 [id=2edd1164c07a4a67b51262d402d53596]). Expected version 3 got 0
I20240503 14:12:28.789503 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
I20240503 14:12:28.790447 28033 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_be0416fa4db54ee0ad44f2faaad25569 [id=2849f6ba9ef64d26a5dc65f9ac9048da] alter complete (version 2)
I20240503 14:12:28.791002 28029 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_ece674e6e74547b6a4d04f1f2b167a08 [id=2edd1164c07a4a67b51262d402d53596] alter complete (version 3)
I20240503 14:12:28.791321 29994 raft_consensus.cc:504] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:28.791906 28029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 113) took 1008 ms (client timeout 1000 ms). Trace:
W20240503 14:12:28.791999 28029 rpcz_store.cc:269] 0503 14:12:27.783362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.789703 (+ 6341us) service_pool.cc:225] Handling call
0503 14:12:28.791877 (+1002174us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.805596 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.805701 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.805706 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:27.805708 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:27.805793 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.805801 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.805806 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:27.805877 (+ 71us) tablet.cc:699] Row locks acquired
0503 14:12:27.805881 (+ 4us) write_op.cc:260] PREPARE: finished
0503 14:12:27.805926 (+ 45us) write_op.cc:270] Start()
0503 14:12:27.805963 (+ 37us) write_op.cc:276] Timestamp: P: 1714745547805890 usec, L: 0
0503 14:12:27.805968 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.806162 (+ 194us) log.cc:842] Serialized 356 byte log entry
0503 14:12:28.658119 (+851957us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.658606 (+ 487us) write_op.cc:301] APPLY: starting
0503 14:12:28.658617 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.658645 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.658667 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.658708 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.658734 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:28.658740 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:28.658792 (+ 52us) log.cc:842] Serialized 23 byte log entry
0503 14:12:28.658832 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.658849 (+ 17us) write_op.cc:454] Released schema lock
0503 14:12:28.658854 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.658905 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.658908 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":457,"apply.run_cpu_time_us":342,"apply.run_wall_time_us":341,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":641,"prepare.run_wall_time_us":1010,"replication_time_us":852138}]]}
I20240503 14:12:28.792021 29994 leader_election.cc:288] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.792903 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_eb6121fb3aa742a3a8aa70b1426d9777" } new_table_name: "default.Table_8668db655be7429dbdc3ee1cd7512979" modify_external_catalogs: true
I20240503 14:12:28.793884 28604 tablet_bootstrap.cc:492] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.794014 28604 ts_tablet_manager.cc:1367] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20240503 14:12:28.794898 28604 raft_consensus.cc:348] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.795099 28604 raft_consensus.cc:374] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.795150 28604 raft_consensus.cc:729] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.795385 28604 consensus_queue.cc:260] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.795663 29996 raft_consensus.cc:482] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.795766 29996 raft_consensus.cc:504] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:28.796310 28033 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 92) took 1009 ms (client timeout 1000 ms). Trace:
I20240503 14:12:28.796388 29996 leader_election.cc:288] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:28.796415 28033 rpcz_store.cc:269] 0503 14:12:27.787110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.792957 (+ 5847us) service_pool.cc:225] Handling call
0503 14:12:28.796284 (+1003327us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:27.803193 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:27.803315 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:27.803323 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:27.803326 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:27.803433 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:27.803446 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:27.803451 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:27.803519 (+ 68us) tablet.cc:699] Row locks acquired
0503 14:12:27.803521 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:27.803558 (+ 37us) write_op.cc:270] Start()
0503 14:12:27.803592 (+ 34us) write_op.cc:276] Timestamp: P: 1714745547803525 usec, L: 0
0503 14:12:27.803596 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:27.803789 (+ 193us) log.cc:842] Serialized 356 byte log entry
0503 14:12:28.658064 (+854275us) op_driver.cc:444] REPLICATION: finished
0503 14:12:28.658180 (+ 116us) write_op.cc:301] APPLY: starting
0503 14:12:28.658192 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:28.658221 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:28.658244 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:28.658290 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:28.658317 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:28.658337 (+ 20us) write_op.cc:312] APPLY: finished
0503 14:12:28.658394 (+ 57us) log.cc:842] Serialized 23 byte log entry
0503 14:12:28.658438 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.658452 (+ 14us) write_op.cc:454] Released schema lock
0503 14:12:28.658457 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:28.658525 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:28.658529 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":390,"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":1094,"replication_time_us":854449}]]}
I20240503 14:12:28.797072 28033 master_service.cc:946] ReplaceTablet: received request to replace tablet 9a1bc119df7d4694bbe926611ef73245 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:28.797178 28604 ts_tablet_manager.cc:1398] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:28.797499 28604 tablet_bootstrap.cc:492] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.799957 28604 tablet_bootstrap.cc:654] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:28.801389 28024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.193:45997 (request call id 104) took 1004 ms (client timeout 1000 ms). Trace:
W20240503 14:12:28.801514 28024 rpcz_store.cc:269] 0503 14:12:27.797216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.617690 (+820474us) service_pool.cc:225] Handling call
0503 14:12:28.659363 (+ 41673us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12ced9668 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:28.801352 (+141989us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":11,"spinlock_wait_cycles":93890304}
I20240503 14:12:28.803057 30418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:28.805485 28661 ts_tablet_manager.cc:1888] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.805785 30103 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.001s
I20240503 14:12:28.806174 30103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
uuid: "bb6a7aee99214df4b02858ddae025b14"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-8wrt"
I20240503 14:12:28.808379 30103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:28.810861 28661 log.cc:1196] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/4e8411f474c247c2abfd0bd9682fb8a2
I20240503 14:12:28.811448 28661 ts_tablet_manager.cc:1909] T 4e8411f474c247c2abfd0bd9682fb8a2 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.812523 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1037) took 1980 ms. Trace:
I20240503 14:12:28.812728 28661 rpcz_store.cc:276] 0503 14:12:26.831882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.832815 (+ 933us) service_pool.cc:225] Handling call
0503 14:12:28.770737 (+1937922us) ts_tablet_manager.cc:1120] Deleting tablet 4e8411f474c247c2abfd0bd9682fb8a2
0503 14:12:28.770751 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.802210 (+ 31459us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.805368 (+ 3158us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.812496 (+ 7128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1937749}
I20240503 14:12:28.813727 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1040) took 1809 ms. Trace:
I20240503 14:12:28.813916 28661 rpcz_store.cc:276] 0503 14:12:27.004438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.021358 (+ 16920us) service_pool.cc:225] Handling call
0503 14:12:28.813570 (+1792212us) ts_tablet_manager.cc:1120] Deleting tablet 5ac145bcc44b4437b840df389cd254f9
0503 14:12:28.813582 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.813706 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790172}
I20240503 14:12:28.814496 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:28.814631 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1039) took 1810 ms. Trace:
I20240503 14:12:28.814721 28212 raft_consensus.cc:3037] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.814889 28604 tablet_bootstrap.cc:492] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.815012 28604 ts_tablet_manager.cc:1367] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 14:12:28.815044 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5189b6cdaa486c8c2f1b5c141d513b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.815223 28213 raft_consensus.cc:2454] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.814704 28661 rpcz_store.cc:276] 0503 14:12:27.004295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.021567 (+ 17272us) service_pool.cc:225] Handling call
0503 14:12:28.814473 (+1792906us) ts_tablet_manager.cc:1120] Deleting tablet 9db657eef2db4e3791318e01e1a965fb
0503 14:12:28.814518 (+ 45us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.814613 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123,"tablet-delete.queue_time_us":1791001}
I20240503 14:12:28.815703 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1042) took 1586 ms. Trace:
I20240503 14:12:28.815771 28604 raft_consensus.cc:348] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.815969 28604 raft_consensus.cc:374] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.816022 28604 raft_consensus.cc:729] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.816241 28604 consensus_queue.cc:260] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.816776 28604 ts_tablet_manager.cc:1398] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:28.817078 28604 tablet_bootstrap.cc:492] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.818786 28212 raft_consensus.cc:2454] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:28.823313 28427 consensus_peers.cc:485] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbcc6cef7b7d464a9fbbae61caf0437d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.823555 28427 leader_election.cc:302] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.824348 29994 raft_consensus.cc:2781] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.824433 29994 raft_consensus.cc:482] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.824491 29994 raft_consensus.cc:3037] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.827235 29994 raft_consensus.cc:504] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.827977 29994 leader_election.cc:288] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:28.828866 28604 tablet_bootstrap.cc:654] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.829782 30315 raft_consensus.cc:482] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.829932 30315 raft_consensus.cc:504] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.830595 30315 leader_election.cc:288] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:28.831696 28427 leader_election.cc:302] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.832201 29994 raft_consensus.cc:2781] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.832753 29994 raft_consensus.cc:686] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:28.833348 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee9b69dd1f47219e7d3126207810c7" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.833110 29994 consensus_queue.cc:237] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.834053 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5189b6cdaa486c8c2f1b5c141d513b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:28.834295 28213 raft_consensus.cc:3037] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.837126 28213 raft_consensus.cc:2454] T ac5189b6cdaa486c8c2f1b5c141d513b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:28.838445 28150 leader_election.cc:341] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.838595 28150 leader_election.cc:302] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.839031 30272 raft_consensus.cc:2726] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.841579 28427 leader_election.cc:302] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.841987 30253 raft_consensus.cc:2781] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.842109 30253 raft_consensus.cc:686] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:28.842376 30253 consensus_queue.cc:237] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.844636 28604 tablet_bootstrap.cc:492] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.844810 28604 ts_tablet_manager.cc:1367] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.000s
I20240503 14:12:28.845672 28604 raft_consensus.cc:348] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.845894 28604 raft_consensus.cc:374] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.845947 28604 raft_consensus.cc:729] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.846148 28604 consensus_queue.cc:260] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.846696 28604 ts_tablet_manager.cc:1398] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:28.846995 28604 tablet_bootstrap.cc:492] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.848768 28604 tablet_bootstrap.cc:654] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.849710 28600 tablet_bootstrap.cc:492] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.849879 28600 ts_tablet_manager.cc:1367] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.171s user 0.002s sys 0.003s
I20240503 14:12:28.850720 28600 raft_consensus.cc:348] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.850950 28600 raft_consensus.cc:374] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.850996 28600 raft_consensus.cc:729] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.851222 28600 consensus_queue.cc:260] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.851822 28600 ts_tablet_manager.cc:1398] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:28.852419 28600 tablet_bootstrap.cc:492] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.854274 28600 tablet_bootstrap.cc:654] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.860163 28604 tablet_bootstrap.cc:492] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.863782 28024 catalog_manager.cc:5484] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:28.864305 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:28.864586 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
I20240503 14:12:28.864830 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2]). Expected version 3 got 0
I20240503 14:12:28.865015 28024 catalog_manager.cc:5484] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:28.865419 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 86d810ab0cf34941a343eef1593632bf (table default.Table_9f2d3cc9c6724dab9ba35ff24e85c236 [id=c358e28088d44cf598c96a1bd2a4321e]). Expected version 1 got 0
I20240503 14:12:28.865664 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:28.865846 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
I20240503 14:12:28.815853 28661 rpcz_store.cc:276] 0503 14:12:27.228922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.229001 (+ 79us) service_pool.cc:225] Handling call
0503 14:12:28.815595 (+1586594us) ts_tablet_manager.cc:1120] Deleting tablet e04c0b6955b34267820ceecfffcfb88a
0503 14:12:28.815604 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.815687 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566611}
I20240503 14:12:28.868188 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1043) took 1639 ms. Trace:
I20240503 14:12:28.870594 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.870764 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5189b6cdaa486c8c2f1b5c141d513b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.871253 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:28.872313 28348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5189b6cdaa486c8c2f1b5c141d513b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:28.872957 28290 leader_election.cc:341] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 02d45788ae644a8581acca54cc55dd3f
I20240503 14:12:28.872066 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d45788ae644a8581acca54cc55dd3f" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:28.873652 28493 raft_consensus.cc:2379] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
W20240503 14:12:28.872756 28427 leader_election.cc:341] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.874210 28661 rpcz_store.cc:276] 0503 14:12:27.229103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.229214 (+ 111us) service_pool.cc:225] Handling call
0503 14:12:28.867998 (+1638784us) ts_tablet_manager.cc:1120] Deleting tablet a526e9f1e9934e519349dcd4523ee694
0503 14:12:28.868012 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.868164 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618911}
I20240503 14:12:28.874557 28661 tablet_replica.cc:330] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.874745 28661 raft_consensus.cc:2229] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.875047 28661 raft_consensus.cc:2258] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.879312 28661 ts_tablet_manager.cc:1875] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.881064 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e87faea1415d95ae014a93fa22af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.881361 30404 tablet_replica.cc:330] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:28.881582 30404 raft_consensus.cc:2229] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.881661 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8097ba5606cf42d68448833de36d85c7) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.881752 30404 raft_consensus.cc:2258] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.882140 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.882344 30404 ts_tablet_manager.cc:1875] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.882607 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b41e3b21347b393497de4d4e87469 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.883163 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.879385 28604 ts_tablet_manager.cc:1367] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.001s
I20240503 14:12:28.879947 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d45788ae644a8581acca54cc55dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:28.880121 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e14e8b5f4046029af4673a277112d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:28.880398 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:28.880537 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8097ba5606cf42d68448833de36d85c7) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:28.883937 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:28.884065 28604 raft_consensus.cc:348] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.884282 28604 raft_consensus.cc:374] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.884330 28604 raft_consensus.cc:729] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.884850 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 129f22db390f4908bcd630623ab4e6b4 (DEFAULT_TABLE table=default.Table_0a2e433bdb54478588ee6b8965e406fc [id=76064dd30db84d24aeed4119c84cc1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.885298 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129f22db390f4908bcd630623ab4e6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.888252 28604 consensus_queue.cc:260] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.888478 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 129f22db390f4908bcd630623ab4e6b4 (DEFAULT_TABLE table=default.Table_0a2e433bdb54478588ee6b8965e406fc [id=76064dd30db84d24aeed4119c84cc1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.888881 28604 ts_tablet_manager.cc:1398] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 14:12:28.888940 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129f22db390f4908bcd630623ab4e6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.889204 28604 tablet_bootstrap.cc:492] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.892736 30404 ts_tablet_manager.cc:1888] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:28.895153 28604 tablet_bootstrap.cc:654] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.896723 30404 log.cc:1196] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/b476e87faea1415d95ae014a93fa22af
I20240503 14:12:28.896494 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 06eeaa67679d443282cdaa7e2d21de33 (DEFAULT_TABLE table=default.Table_45c301ad86394025a523ce52461c5606 [id=70fef4d9e24d46f29b4ba45bbcaff60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.897379 30404 ts_tablet_manager.cc:1909] T b476e87faea1415d95ae014a93fa22af P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:28.897359 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06eeaa67679d443282cdaa7e2d21de33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.898306 30404 tablet_replica.cc:330] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:28.898488 30404 raft_consensus.cc:2229] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.898805 30404 raft_consensus.cc:2258] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.900033 28600 tablet_bootstrap.cc:492] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.900192 28600 ts_tablet_manager.cc:1367] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.002s
I20240503 14:12:28.900826 28614 tablet_bootstrap.cc:492] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:28.900966 28614 ts_tablet_manager.cc:1367] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.177s user 0.003s sys 0.001s
I20240503 14:12:28.901769 28600 raft_consensus.cc:348] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.901965 28600 raft_consensus.cc:374] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.902007 28600 raft_consensus.cc:729] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.902199 28600 consensus_queue.cc:260] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.902957 28600 ts_tablet_manager.cc:1398] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:28.903321 28600 tablet_bootstrap.cc:492] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.905508 30404 ts_tablet_manager.cc:1875] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.901746 28614 raft_consensus.cc:348] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.908756 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 06eeaa67679d443282cdaa7e2d21de33 (DEFAULT_TABLE table=default.Table_45c301ad86394025a523ce52461c5606 [id=70fef4d9e24d46f29b4ba45bbcaff60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.909061 28614 raft_consensus.cc:374] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.909155 28614 raft_consensus.cc:729] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.909293 28661 ts_tablet_manager.cc:1888] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.909430 28604 tablet_bootstrap.cc:492] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:28.909451 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06eeaa67679d443282cdaa7e2d21de33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.909423 28614 consensus_queue.cc:260] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.909550 28661 log.cc:1196] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a884cf4ea11e48ef941722b79ad50075
I20240503 14:12:28.906921 28600 tablet_bootstrap.cc:654] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.909583 28604 ts_tablet_manager.cc:1367] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240503 14:12:28.910881 30253 raft_consensus.cc:482] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:28.911150 30253 raft_consensus.cc:504] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.911496 28661 ts_tablet_manager.cc:1909] T a884cf4ea11e48ef941722b79ad50075 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.911875 30253 leader_election.cc:288] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:28.912446 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b2865e9604ce58280321c7ee2a88e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:28.912701 28212 raft_consensus.cc:2454] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.912689 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1044) took 1597 ms. Trace:
I20240503 14:12:28.912776 28661 rpcz_store.cc:276] 0503 14:12:27.315135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.315195 (+ 60us) service_pool.cc:225] Handling call
0503 14:12:28.874505 (+1559310us) ts_tablet_manager.cc:1120] Deleting tablet a884cf4ea11e48ef941722b79ad50075
0503 14:12:28.874516 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.898571 (+ 24055us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.909172 (+ 10601us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.912665 (+ 3493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546427}
I20240503 14:12:28.913262 28427 leader_election.cc:302] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.913305 28614 ts_tablet_manager.cc:1398] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 14:12:28.913532 30253 raft_consensus.cc:2781] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:28.913600 30253 raft_consensus.cc:482] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:28.913656 30253 raft_consensus.cc:3037] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.913654 28614 tablet_bootstrap.cc:492] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:28.915871 29828 raft_consensus.cc:2229] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:28.916146 29828 raft_consensus.cc:2258] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.916483 30253 raft_consensus.cc:504] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:28.916734 29828 ts_tablet_manager.cc:1875] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.917145 30253 leader_election.cc:288] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:28.918001 28661 tablet_replica.cc:330] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.918210 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b2865e9604ce58280321c7ee2a88e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:28.918282 28661 raft_consensus.cc:2229] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.918434 28212 raft_consensus.cc:3037] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.918352 28604 raft_consensus.cc:348] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.918509 28604 raft_consensus.cc:374] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.918555 28604 raft_consensus.cc:729] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.918650 28661 raft_consensus.cc:2258] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.918766 28604 consensus_queue.cc:260] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.919816 28661 ts_tablet_manager.cc:1875] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.921058 28212 raft_consensus.cc:2454] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:28.921340 28614 tablet_bootstrap.cc:654] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.921478 28427 leader_election.cc:302] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:28.921766 30253 raft_consensus.cc:2781] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:28.922425 30253 raft_consensus.cc:686] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:28.924829 28604 ts_tablet_manager.cc:1398] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:12:28.925202 28604 tablet_bootstrap.cc:492] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:28.931948 28661 ts_tablet_manager.cc:1888] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.933619 28600 tablet_bootstrap.cc:492] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.933872 28600 ts_tablet_manager.cc:1367] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.003s
I20240503 14:12:28.933876 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3e7b2865e9604ce58280321c7ee2a88e (table default.Table_ca6cee392d934dd3ac773e100ae031d2 [id=32be054076954aed9265888ca62e510b]). Expected version 4 got 0
I20240503 14:12:28.934532 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (table default.Table_bfff85704d88463383cbf94f9b121359 [id=bd1950deb4274d86bf288d501cf5c710]). Expected version 2 got 0
I20240503 14:12:28.934837 28600 raft_consensus.cc:348] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.935160 28600 raft_consensus.cc:374] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.934965 28024 catalog_manager.cc:5484] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:28.935323 28600 raft_consensus.cc:729] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.935618 28600 consensus_queue.cc:260] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.936473 28600 ts_tablet_manager.cc:1398] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:28.936789 28600 tablet_bootstrap.cc:492] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.938216 28187 tablet_service.cc:1467] Processing CreateTablet for tablet ca74f89636914266a5d9c9d93d3e5d63 (DEFAULT_TABLE table=default.Table_7aa3d349d8e24199ada0764815b8d96e [id=142041e8607541b8bfa097e81d223c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.938736 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca74f89636914266a5d9c9d93d3e5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.938807 28600 tablet_bootstrap.cc:654] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.939949 28661 log.cc:1196] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/c9325d5142af4b018354bf38370b61e0
I20240503 14:12:28.940450 28661 ts_tablet_manager.cc:1909] T c9325d5142af4b018354bf38370b61e0 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:28.940840 30404 ts_tablet_manager.cc:1888] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:28.940960 30404 log.cc:1196] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/c816f3b05cfb4496984f6cb28d310cc4
I20240503 14:12:28.941298 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1045) took 1625 ms. Trace:
I20240503 14:12:28.941397 28661 rpcz_store.cc:276] 0503 14:12:27.315470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.315521 (+ 51us) service_pool.cc:225] Handling call
0503 14:12:28.917936 (+1602415us) ts_tablet_manager.cc:1120] Deleting tablet c9325d5142af4b018354bf38370b61e0
0503 14:12:28.917950 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.928843 (+ 10893us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.931828 (+ 2985us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:28.941270 (+ 9442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1589764}
I20240503 14:12:28.941828 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1046) took 1521 ms. Trace:
W20240503 14:12:28.941831 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 68695.3us
I20240503 14:12:28.941906 28661 rpcz_store.cc:276] 0503 14:12:27.420171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.420227 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:28.941692 (+1521465us) ts_tablet_manager.cc:1120] Deleting tablet b97da778da214bb6bc147c5ada6c7603
0503 14:12:28.941702 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.941813 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1521194}
I20240503 14:12:28.942178 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1047) took 1521 ms. Trace:
I20240503 14:12:28.942261 28661 rpcz_store.cc:276] 0503 14:12:27.421159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.421211 (+ 52us) service_pool.cc:225] Handling call
0503 14:12:28.942094 (+1520883us) ts_tablet_manager.cc:1120] Deleting tablet f91ddbc250c44fbfbf4ffe83433d8dd8
0503 14:12:28.942102 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.942165 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1520678}
I20240503 14:12:28.942592 28661 tablet_replica.cc:330] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:28.942790 28661 raft_consensus.cc:2229] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:28.943075 28661 pending_rounds.cc:62] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:28.943131 28661 pending_rounds.cc:68] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597763886944256 op_type: NO_OP noop_request { }
I20240503 14:12:28.943251 28661 raft_consensus.cc:2866] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:28.943334 28661 raft_consensus.cc:2258] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:28.944957 28661 ts_tablet_manager.cc:1875] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:28.946775 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b2865e9604ce58280321c7ee2a88e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:28.947019 28355 raft_consensus.cc:2454] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:28.947290 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b2865e9604ce58280321c7ee2a88e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:28.947456 28352 raft_consensus.cc:3037] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.947746 28290 leader_election.cc:302] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:28.948454 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 129f22db390f4908bcd630623ab4e6b4 (DEFAULT_TABLE table=default.Table_0a2e433bdb54478588ee6b8965e406fc [id=76064dd30db84d24aeed4119c84cc1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.948679 28600 tablet_bootstrap.cc:492] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.948823 28600 ts_tablet_manager.cc:1367] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240503 14:12:28.948900 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129f22db390f4908bcd630623ab4e6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.949594 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 06eeaa67679d443282cdaa7e2d21de33 (DEFAULT_TABLE table=default.Table_45c301ad86394025a523ce52461c5606 [id=70fef4d9e24d46f29b4ba45bbcaff60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.949719 28600 raft_consensus.cc:348] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.949911 28600 raft_consensus.cc:374] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.949998 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06eeaa67679d443282cdaa7e2d21de33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.949996 28600 raft_consensus.cc:729] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.950265 28600 consensus_queue.cc:260] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.950789 28335 tablet_service.cc:1467] Processing CreateTablet for tablet ca74f89636914266a5d9c9d93d3e5d63 (DEFAULT_TABLE table=default.Table_7aa3d349d8e24199ada0764815b8d96e [id=142041e8607541b8bfa097e81d223c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.951036 28600 ts_tablet_manager.cc:1398] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:28.951109 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca74f89636914266a5d9c9d93d3e5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.951355 28600 tablet_bootstrap.cc:492] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.952684 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2fa8fa721542e9bb814b405b8e6208 (DEFAULT_TABLE table=default.Table_cd02de883e9a43a79775ffc0e5deb283 [id=d8ad2abd71954f60a12fc3e396813bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.953114 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2fa8fa721542e9bb814b405b8e6208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.953240 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7fbc0c4cd843efad4fe098e335303e (DEFAULT_TABLE table=default.Table_4cd19da5bf64494e8ff53ded46bc5c4b [id=8adf0b81e1e1448fa0ef8a19c2364eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.953342 28600 tablet_bootstrap.cc:654] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.953600 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7fbc0c4cd843efad4fe098e335303e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.955966 28600 tablet_bootstrap.cc:492] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.956195 28600 ts_tablet_manager.cc:1367] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:28.956498 28188 tablet_service.cc:1467] Processing CreateTablet for tablet f87b9edf09814893997ab4578dfe6ee5 (DEFAULT_TABLE table=default.Table_e6e71c3cb9d7443c8354b100cd955068 [id=82dd7ed387914f519c6a8e751c4fa2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.956877 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87b9edf09814893997ab4578dfe6ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.957201 28600 raft_consensus.cc:348] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.957479 28600 raft_consensus.cc:374] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.957589 28600 raft_consensus.cc:729] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.957890 28600 consensus_queue.cc:260] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.958727 28600 ts_tablet_manager.cc:1398] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:28.958859 30431 raft_consensus.cc:3037] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:28.959077 28600 tablet_bootstrap.cc:492] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.959921 28024 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 34238c9859724d97a32f2a1d9315edb1 (table default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]). Expected version 1 got 0
W20240503 14:12:28.960381 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e220336186f648419b186feada6f0124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e220336186f648419b186feada6f0124 already in progress: creating tablet
I20240503 14:12:28.960435 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 6518f93f7a5c444aa91c9473a5111e5b (DEFAULT_TABLE table=default.Table_a6fe66b4dbee47e9b0b15b491131bfe7 [id=f004db5dd3b745ab8a634efe92efb51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.960814 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6518f93f7a5c444aa91c9473a5111e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.960777 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet b476e87faea1415d95ae014a93fa22af (table default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]) successfully deleted
I20240503 14:12:28.961078 28600 tablet_bootstrap.cc:654] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.961225 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 20b176a071ba4f3fbff2a64d6e46027a (table default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]) successfully deleted
I20240503 14:12:28.961419 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 4e8411f474c247c2abfd0bd9682fb8a2 (table default.Table_fc189ad8c19041f7814fac6de9f482cb [id=b7f4f49135e245dc975ffd7eb0589a8d]) successfully deleted
W20240503 14:12:28.961622 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5ac145bcc44b4437b840df389cd254f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ac145bcc44b4437b840df389cd254f9 already in progress: creating tablet
W20240503 14:12:28.961786 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9db657eef2db4e3791318e01e1a965fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9db657eef2db4e3791318e01e1a965fb already in progress: creating tablet
W20240503 14:12:28.961970 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e04c0b6955b34267820ceecfffcfb88a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e04c0b6955b34267820ceecfffcfb88a already in progress: creating tablet
W20240503 14:12:28.962138 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a526e9f1e9934e519349dcd4523ee694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a526e9f1e9934e519349dcd4523ee694 already in progress: creating tablet
I20240503 14:12:28.962399 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a884cf4ea11e48ef941722b79ad50075 (table default.Table_cc77503fb1ad4d03a6278aef9f3ab979 [id=4b3b8bed2fba4370a246902bec7554f7]) successfully deleted
I20240503 14:12:28.963065 30103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:28.963474 28600 tablet_bootstrap.cc:492] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.963630 28600 ts_tablet_manager.cc:1367] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:28.964457 28600 raft_consensus.cc:348] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.964707 28600 raft_consensus.cc:374] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.964792 28600 raft_consensus.cc:729] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.965037 28600 consensus_queue.cc:260] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.965888 28600 ts_tablet_manager.cc:1398] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:28.966215 28600 tablet_bootstrap.cc:492] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.968362 28600 tablet_bootstrap.cc:654] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:28.968716 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 328bb280f36d4c79b2795c10df2b836b (DEFAULT_TABLE table=default.Table_2bc25daca38a439a9c3275f9cb988224 [id=33e1fdb6383a46f884cae02c5d512f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.968937 28322 tablet_service.cc:1467] Processing CreateTablet for tablet cfe4baf20af348459c229e61fe6d73d5 (DEFAULT_TABLE table=default.Table_af7c6d0b47e24c60b73741ec9e1f6d7b [id=8a61faf53b6a486a921a1b7c55561265]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.969135 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328bb280f36d4c79b2795c10df2b836b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.969321 30103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:28.969497 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2fa8fa721542e9bb814b405b8e6208 (DEFAULT_TABLE table=default.Table_cd02de883e9a43a79775ffc0e5deb283 [id=d8ad2abd71954f60a12fc3e396813bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.970402 30103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:28.947463 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:28.975553 28352 raft_consensus.cc:2454] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:28.978197 28600 tablet_bootstrap.cc:492] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:28.978422 28600 ts_tablet_manager.cc:1367] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240503 14:12:28.979374 28600 raft_consensus.cc:348] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.979647 28600 raft_consensus.cc:374] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:28.979740 28600 raft_consensus.cc:729] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.979712 28197 tablet_service.cc:1467] Processing CreateTablet for tablet cfe4baf20af348459c229e61fe6d73d5 (DEFAULT_TABLE table=default.Table_af7c6d0b47e24c60b73741ec9e1f6d7b [id=8a61faf53b6a486a921a1b7c55561265]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.980269 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe4baf20af348459c229e61fe6d73d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.980165 28600 consensus_queue.cc:260] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:28.980950 28600 ts_tablet_manager.cc:1398] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:28.981261 28600 tablet_bootstrap.cc:492] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:28.982365 28330 tablet_service.cc:1467] Processing CreateTablet for tablet f87b9edf09814893997ab4578dfe6ee5 (DEFAULT_TABLE table=default.Table_e6e71c3cb9d7443c8354b100cd955068 [id=82dd7ed387914f519c6a8e751c4fa2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.983179 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 6518f93f7a5c444aa91c9473a5111e5b (DEFAULT_TABLE table=default.Table_a6fe66b4dbee47e9b0b15b491131bfe7 [id=f004db5dd3b745ab8a634efe92efb51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.983290 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87b9edf09814893997ab4578dfe6ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.983505 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe4baf20af348459c229e61fe6d73d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.986861 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6518f93f7a5c444aa91c9473a5111e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.988277 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2fa8fa721542e9bb814b405b8e6208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.988349 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 328bb280f36d4c79b2795c10df2b836b (DEFAULT_TABLE table=default.Table_2bc25daca38a439a9c3275f9cb988224 [id=33e1fdb6383a46f884cae02c5d512f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.988745 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328bb280f36d4c79b2795c10df2b836b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.989549 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7fbc0c4cd843efad4fe098e335303e (DEFAULT_TABLE table=default.Table_4cd19da5bf64494e8ff53ded46bc5c4b [id=8adf0b81e1e1448fa0ef8a19c2364eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:28.989903 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7fbc0c4cd843efad4fe098e335303e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.993145 30431 raft_consensus.cc:2726] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:28.999495 28600 tablet_bootstrap.cc:654] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.008507 28600 tablet_bootstrap.cc:492] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.008738 28600 ts_tablet_manager.cc:1367] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.006s
I20240503 14:12:29.009032 29828 ts_tablet_manager.cc:1888] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.009137 29828 log.cc:1196] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7b6c998e518541248a87e4c1551ec080
I20240503 14:12:29.009398 29828 ts_tablet_manager.cc:1909] T 7b6c998e518541248a87e4c1551ec080 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.009713 28600 raft_consensus.cc:348] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.009974 28600 raft_consensus.cc:374] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.011982 28600 raft_consensus.cc:729] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.012336 28600 consensus_queue.cc:260] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.013665 28600 ts_tablet_manager.cc:1398] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:12:29.014597 28600 tablet_bootstrap.cc:492] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:29.016155 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 53514.7us
I20240503 14:12:29.016954 28600 tablet_bootstrap.cc:654] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.019493 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 980) took 3001 ms. Trace:
I20240503 14:12:29.020439 29828 rpcz_store.cc:276] 0503 14:12:26.017562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.017600 (+ 38us) service_pool.cc:225] Handling call
0503 14:12:28.766592 (+2748992us) ts_tablet_manager.cc:1120] Deleting tablet 7b6c998e518541248a87e4c1551ec080
0503 14:12:28.766603 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:28.987629 (+221026us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.008919 (+ 21290us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.019457 (+ 10538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2748860}
I20240503 14:12:29.022452 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 981) took 3004 ms. Trace:
I20240503 14:12:29.023175 29828 rpcz_store.cc:276] 0503 14:12:26.017855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.017894 (+ 39us) service_pool.cc:225] Handling call
0503 14:12:29.022310 (+3004416us) ts_tablet_manager.cc:1120] Deleting tablet ed4dc732bcb44222b4b15e5d77defe33
0503 14:12:29.022323 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.022429 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004285}
I20240503 14:12:29.023873 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 982) took 3005 ms. Trace:
I20240503 14:12:29.024267 29828 rpcz_store.cc:276] 0503 14:12:26.018131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.018167 (+ 36us) service_pool.cc:225] Handling call
0503 14:12:29.023713 (+3005546us) ts_tablet_manager.cc:1120] Deleting tablet 6a27d21de6a6441d810f379bba88f9a2
0503 14:12:29.023725 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.023853 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005418}
I20240503 14:12:29.027345 28187 tablet_service.cc:1467] Processing CreateTablet for tablet d0912772bb7446cb9e8f796d24a02225 (DEFAULT_TABLE table=default.Table_7551243835a942e4ac44d0f728b28fe5 [id=6a8e3968b93f4784bc9cc6ac96db8390]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.027953 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0912772bb7446cb9e8f796d24a02225. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.030151 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 989) took 2942 ms. Trace:
I20240503 14:12:29.030095 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 61da48859c4d4a73bb1b9d7c0ede8106 (DEFAULT_TABLE table=default.Table_c9fcd1ad02784a29858da8e74a9c9866 [id=38c77a1688424f44b3aeeb9384b445bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.030630 29828 rpcz_store.cc:276] 0503 14:12:26.088182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.088238 (+ 56us) service_pool.cc:225] Handling call
0503 14:12:29.029965 (+2941727us) ts_tablet_manager.cc:1120] Deleting tablet 5de9edfb0d064939bda2ec7a9e0e500b
0503 14:12:29.029981 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.030125 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2941405}
I20240503 14:12:29.031227 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61da48859c4d4a73bb1b9d7c0ede8106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.031333 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 992) took 2742 ms. Trace:
I20240503 14:12:29.032498 30272 raft_consensus.cc:482] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.032701 30272 raft_consensus.cc:504] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.033478 30272 leader_election.cc:288] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.035006 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b21cd1262d94d1c994b29c44f86cb5e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.035246 28427 leader_election.cc:341] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.035719 28427 leader_election.cc:341] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.035723 28150 leader_election.cc:341] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:28.922700 30253 consensus_queue.cc:237] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.036154 29828 rpcz_store.cc:276] 0503 14:12:26.288866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.288910 (+ 44us) service_pool.cc:225] Handling call
0503 14:12:29.031185 (+2742275us) ts_tablet_manager.cc:1120] Deleting tablet 13f8553cfbab4af2a38a73e81fed2a5f
0503 14:12:29.031197 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.031313 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2727745}
I20240503 14:12:29.036594 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 993) took 2747 ms. Trace:
I20240503 14:12:29.036669 29828 rpcz_store.cc:276] 0503 14:12:26.289106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.289148 (+ 42us) service_pool.cc:225] Handling call
0503 14:12:29.036459 (+2747311us) ts_tablet_manager.cc:1120] Deleting tablet 9f5b973ea16d42babd2cc75e32624854
0503 14:12:29.036471 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.036577 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2732957}
I20240503 14:12:29.036957 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 991) took 2748 ms. Trace:
I20240503 14:12:29.037012 29828 rpcz_store.cc:276] 0503 14:12:26.288548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.288595 (+ 47us) service_pool.cc:225] Handling call
0503 14:12:29.036869 (+2748274us) ts_tablet_manager.cc:1120] Deleting tablet 19d43cc864064c4c907ec0629391da10
0503 14:12:29.036877 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.036945 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2728709}
I20240503 14:12:29.037350 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 990) took 2815 ms. Trace:
I20240503 14:12:29.037418 29828 rpcz_store.cc:276] 0503 14:12:26.288395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.289938 (+ 1543us) service_pool.cc:225] Handling call
0503 14:12:29.037249 (+2747311us) ts_tablet_manager.cc:1120] Deleting tablet 0d4c5648ddc84407a8d8b44f2d63dcee
0503 14:12:29.037259 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.037333 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2713619}
I20240503 14:12:29.037683 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1002) took 2637 ms. Trace:
I20240503 14:12:29.037756 29828 rpcz_store.cc:276] 0503 14:12:26.399774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.410701 (+ 10927us) service_pool.cc:225] Handling call
0503 14:12:29.037590 (+2626889us) ts_tablet_manager.cc:1120] Deleting tablet addc2a6195034b0a927fdee7a2955122
0503 14:12:29.037598 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.037669 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2626553}
I20240503 14:12:29.038033 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1003) took 2552 ms. Trace:
I20240503 14:12:29.038138 29828 rpcz_store.cc:276] 0503 14:12:26.485683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.488158 (+ 2475us) service_pool.cc:225] Handling call
0503 14:12:29.037943 (+2549785us) ts_tablet_manager.cc:1120] Deleting tablet 77511682d71a40f5933edcb6a6523649
0503 14:12:29.037951 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.038019 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2474728}
I20240503 14:12:29.038472 29828 tablet_replica.cc:330] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.034639 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b21cd1262d94d1c994b29c44f86cb5e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.038668 28188 tablet_service.cc:1467] Processing CreateTablet for tablet a0f4738ccaad47ab87317ee90efcd46c (DEFAULT_TABLE table=default.Table_70df8ea2254f45aab971123dd7f55a44 [id=24611f8249494aa094924164a4f2c1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.039196 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f4738ccaad47ab87317ee90efcd46c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:29.039258 28150 leader_election.cc:341] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.039412 28150 leader_election.cc:302] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.039712 30272 raft_consensus.cc:2726] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.040167 29828 raft_consensus.cc:2229] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.042953 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 61da48859c4d4a73bb1b9d7c0ede8106 (DEFAULT_TABLE table=default.Table_c9fcd1ad02784a29858da8e74a9c9866 [id=38c77a1688424f44b3aeeb9384b445bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.043854 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61da48859c4d4a73bb1b9d7c0ede8106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.044653 28335 tablet_service.cc:1467] Processing CreateTablet for tablet eff8442b6c02469eaedb08c51200829b (DEFAULT_TABLE table=default.Table_3d0d6e04014b414aa95434036f1b4516 [id=dd13763c434241bb9352dd30d072c5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.044814 28661 ts_tablet_manager.cc:1888] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.044951 28661 log.cc:1196] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/dbcc6cef7b7d464a9fbbae61caf0437d
I20240503 14:12:29.045292 28661 ts_tablet_manager.cc:1909] T dbcc6cef7b7d464a9fbbae61caf0437d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.045317 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff8442b6c02469eaedb08c51200829b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.046110 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1065) took 1103 ms. Trace:
I20240503 14:12:29.046213 28661 rpcz_store.cc:276] 0503 14:12:27.942223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.945404 (+ 3181us) service_pool.cc:225] Handling call
0503 14:12:28.942464 (+997060us) ts_tablet_manager.cc:1120] Deleting tablet dbcc6cef7b7d464a9fbbae61caf0437d
0503 14:12:28.942472 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.037419 (+ 94947us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.044682 (+ 7263us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.046082 (+ 1400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":996880}
I20240503 14:12:29.046548 28661 tablet_replica.cc:330] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.046761 28661 raft_consensus.cc:2229] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:29.047142 28661 raft_consensus.cc:2258] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.048388 28661 ts_tablet_manager.cc:1875] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.052352 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 700bc4204db84371bd21f17eefc24521 (DEFAULT_TABLE table=default.Table_1c8ad5b7f0304d83a8cdf65bff9e76e5 [id=f1dc41026a4a421f95b9d6903a2700b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.052889 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700bc4204db84371bd21f17eefc24521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.053030 28604 tablet_bootstrap.cc:654] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.057657 28661 ts_tablet_manager.cc:1888] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.057835 28661 log.cc:1196] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/3d54cc5c03034a6685c1da34f9bd65ac
I20240503 14:12:29.058197 28661 ts_tablet_manager.cc:1909] T 3d54cc5c03034a6685c1da34f9bd65ac P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.059069 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1066) took 1113 ms. Trace:
I20240503 14:12:29.059238 30272 raft_consensus.cc:482] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.059427 30272 raft_consensus.cc:504] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.060642 30272 leader_election.cc:288] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.060637 30253 raft_consensus.cc:482] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.060783 30253 raft_consensus.cc:504] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.061412 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.062099 28150 leader_election.cc:341] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.061424 30253 leader_election.cc:288] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.059176 28661 rpcz_store.cc:276] 0503 14:12:27.945774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.945816 (+ 42us) service_pool.cc:225] Handling call
0503 14:12:29.046499 (+1100683us) ts_tablet_manager.cc:1120] Deleting tablet 3d54cc5c03034a6685c1da34f9bd65ac
0503 14:12:29.046510 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.051701 (+ 5191us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.057535 (+ 5834us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.059041 (+ 1506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100543}
I20240503 14:12:29.064616 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1067) took 1118 ms. Trace:
I20240503 14:12:29.064846 30272 raft_consensus.cc:482] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.065035 30272 raft_consensus.cc:504] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.065182 28661 rpcz_store.cc:276] 0503 14:12:27.946055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.946089 (+ 34us) service_pool.cc:225] Handling call
0503 14:12:29.064467 (+1118378us) ts_tablet_manager.cc:1120] Deleting tablet 5500da0e5658492b840dd29273ab265f
0503 14:12:29.064479 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.064593 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118228}
I20240503 14:12:29.065760 30272 leader_election.cc:288] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.066145 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1068) took 1119 ms. Trace:
I20240503 14:12:29.066354 28186 tablet_service.cc:1467] Processing CreateTablet for tablet eff8442b6c02469eaedb08c51200829b (DEFAULT_TABLE table=default.Table_3d0d6e04014b414aa95434036f1b4516 [id=dd13763c434241bb9352dd30d072c5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.066553 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6696a237c3dd428097e2ebdf24afd63f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.066488 28661 rpcz_store.cc:276] 0503 14:12:27.946387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.946423 (+ 36us) service_pool.cc:225] Handling call
0503 14:12:29.065825 (+1119402us) ts_tablet_manager.cc:1120] Deleting tablet 5685a6b3ea6943f8a72d74de87ac2166
0503 14:12:29.065836 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.066125 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119253}
I20240503 14:12:29.067050 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff8442b6c02469eaedb08c51200829b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.067087 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1060) took 1125 ms. Trace:
W20240503 14:12:29.067203 28150 leader_election.cc:341] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.067169 28661 rpcz_store.cc:276] 0503 14:12:27.941844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.955202 (+ 13358us) service_pool.cc:225] Handling call
0503 14:12:29.066978 (+1111776us) ts_tablet_manager.cc:1120] Deleting tablet 3dfd6d40a57e4bea973874ff12c0ea3a
0503 14:12:29.066988 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.067071 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111557}
I20240503 14:12:29.067420 28661 tablet_replica.cc:330] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.067633 28661 raft_consensus.cc:2229] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.067802 28661 raft_consensus.cc:2258] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.068135 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6dbc568abd49d896739a36717b993b (DEFAULT_TABLE table=default.Table_9d113544122942a38f6c98e899f8c488 [id=49aafc9bf6624c2b9363fd18ccd242ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.068423 28326 tablet_service.cc:1467] Processing CreateTablet for tablet d0912772bb7446cb9e8f796d24a02225 (DEFAULT_TABLE table=default.Table_7551243835a942e4ac44d0f728b28fe5 [id=6a8e3968b93f4784bc9cc6ac96db8390]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.068472 28661 ts_tablet_manager.cc:1875] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.068578 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6dbc568abd49d896739a36717b993b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.069098 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 700bc4204db84371bd21f17eefc24521 (DEFAULT_TABLE table=default.Table_1c8ad5b7f0304d83a8cdf65bff9e76e5 [id=f1dc41026a4a421f95b9d6903a2700b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.069551 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700bc4204db84371bd21f17eefc24521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.072250 30272 raft_consensus.cc:482] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.072564 30272 raft_consensus.cc:504] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.073408 30272 leader_election.cc:288] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.074334 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d348da459634561985c4ac1710c0bb2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.074574 28493 raft_consensus.cc:2454] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:29.075246 28150 leader_election.cc:302] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.075400 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6dbc568abd49d896739a36717b993b (DEFAULT_TABLE table=default.Table_9d113544122942a38f6c98e899f8c488 [id=49aafc9bf6624c2b9363fd18ccd242ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.075771 30272 raft_consensus.cc:2781] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.075966 30272 raft_consensus.cc:482] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.076181 30272 raft_consensus.cc:3037] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.078397 28661 ts_tablet_manager.cc:1888] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.078536 28661 log.cc:1196] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a3d1314841574fc8bcf3ccb16755e421
I20240503 14:12:29.078804 28661 ts_tablet_manager.cc:1909] T a3d1314841574fc8bcf3ccb16755e421 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.079017 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0912772bb7446cb9e8f796d24a02225. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.079447 30272 raft_consensus.cc:504] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.079617 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1061) took 1137 ms. Trace:
I20240503 14:12:29.079717 28661 rpcz_store.cc:276] 0503 14:12:27.941923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.955466 (+ 13543us) service_pool.cc:225] Handling call
0503 14:12:29.067374 (+1111908us) ts_tablet_manager.cc:1120] Deleting tablet a3d1314841574fc8bcf3ccb16755e421
0503 14:12:29.067382 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.075290 (+ 7908us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.078280 (+ 2990us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.079588 (+ 1308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111755}
I20240503 14:12:29.080142 28661 tablet_replica.cc:330] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.080289 29828 raft_consensus.cc:2258] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.080353 28661 raft_consensus.cc:2229] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:29.080687 28661 raft_consensus.cc:2258] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.081146 29828 ts_tablet_manager.cc:1875] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.081493 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d348da459634561985c4ac1710c0bb2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:29.081771 28493 raft_consensus.cc:3037] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.082326 28321 tablet_service.cc:1467] Processing CreateTablet for tablet a0f4738ccaad47ab87317ee90efcd46c (DEFAULT_TABLE table=default.Table_70df8ea2254f45aab971123dd7f55a44 [id=24611f8249494aa094924164a4f2c1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.075851 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6dbc568abd49d896739a36717b993b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.084897 28493 raft_consensus.cc:2454] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:29.085561 28150 leader_election.cc:302] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.085930 30266 raft_consensus.cc:2781] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.092896 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f4738ccaad47ab87317ee90efcd46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.095713 30266 raft_consensus.cc:686] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:29.099627 28322 tablet_service.cc:1467] Processing CreateTablet for tablet d453860bc12d48efaeafd8bf781859b5 (DEFAULT_TABLE table=default.Table_6a7863d34f3a47f38c253755f3089f6e [id=fadbf65b20be403e8ba3594bc9a95c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.099826 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 655758ecce9a4d01bf36d4d846436b7b (DEFAULT_TABLE table=default.Table_2300b3e81cff4e629ac39007ac3ba4af [id=32e6719c7fae43d28a5cff49f40d93e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.100129 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d453860bc12d48efaeafd8bf781859b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.100853 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655758ecce9a4d01bf36d4d846436b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.102962 30431 raft_consensus.cc:482] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.103128 30431 raft_consensus.cc:504] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.103845 30431 leader_election.cc:288] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.104193 30253 raft_consensus.cc:482] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.104346 30253 raft_consensus.cc:504] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.104496 28337 tablet_service.cc:1467] Processing CreateTablet for tablet f53643c4a69648989a4bd0c30c5dd24f (DEFAULT_TABLE table=default.Table_2fe7f4a5bc6f467e890e502b73d2545c [id=8631dcb5087e4ded87da93fc930c2031]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.104957 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53643c4a69648989a4bd0c30c5dd24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.104987 30253 leader_election.cc:288] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.108903 28614 tablet_bootstrap.cc:492] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.109107 28614 ts_tablet_manager.cc:1367] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.196s user 0.001s sys 0.004s
I20240503 14:12:29.109306 30448 raft_consensus.cc:482] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.109546 30448 raft_consensus.cc:504] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.111085 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf810e84367f420b928a171790849445" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.111411 28493 raft_consensus.cc:2454] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:29.112879 30448 leader_election.cc:288] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.113143 30272 leader_election.cc:288] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.113674 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7b6c998e518541248a87e4c1551ec080 (table default.Table_a7aefabd0c3841618e68382b8fa41a9a [id=c23a0b9e98f949559497aea2a4dab24a]) successfully deleted
I20240503 14:12:29.113845 28197 tablet_service.cc:1467] Processing CreateTablet for tablet d453860bc12d48efaeafd8bf781859b5 (DEFAULT_TABLE table=default.Table_6a7863d34f3a47f38c253755f3089f6e [id=fadbf65b20be403e8ba3594bc9a95c83]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:29.114297 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ed4dc732bcb44222b4b15e5d77defe33 because the tablet was not found. No further retry: Not found: Tablet not found: ed4dc732bcb44222b4b15e5d77defe33
I20240503 14:12:29.114472 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 655758ecce9a4d01bf36d4d846436b7b (DEFAULT_TABLE table=default.Table_2300b3e81cff4e629ac39007ac3ba4af [id=32e6719c7fae43d28a5cff49f40d93e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.114482 30266 consensus_queue.cc:237] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.121102 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c7a51c68ef478e9bdc5b3e3dc3d5df" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.121474 28493 raft_consensus.cc:2454] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:29.125257 29828 ts_tablet_manager.cc:1888] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.128191 29828 log.cc:1196] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/11ac912a3bd64cdc9082596e09c72e13
I20240503 14:12:29.129177 29828 ts_tablet_manager.cc:1909] T 11ac912a3bd64cdc9082596e09c72e13 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:29.129417 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 92231.9us
W20240503 14:12:29.129699 28427 leader_election.cc:341] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.130671 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1007) took 2573 ms. Trace:
W20240503 14:12:29.132500 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6a27d21de6a6441d810f379bba88f9a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a27d21de6a6441d810f379bba88f9a2 already in progress: creating tablet
W20240503 14:12:29.133080 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5de9edfb0d064939bda2ec7a9e0e500b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9edfb0d064939bda2ec7a9e0e500b already in progress: creating tablet
W20240503 14:12:29.133728 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 13f8553cfbab4af2a38a73e81fed2a5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13f8553cfbab4af2a38a73e81fed2a5f already in progress: creating tablet
W20240503 14:12:29.134172 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9f5b973ea16d42babd2cc75e32624854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f5b973ea16d42babd2cc75e32624854 already in progress: creating tablet
I20240503 14:12:29.134011 29828 rpcz_store.cc:276] 0503 14:12:26.556774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.557182 (+ 408us) service_pool.cc:225] Handling call
0503 14:12:29.038431 (+2481249us) ts_tablet_manager.cc:1120] Deleting tablet 11ac912a3bd64cdc9082596e09c72e13
0503 14:12:29.038439 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.100999 (+ 62560us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.125134 (+ 24135us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.130639 (+ 5505us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39552,"tablet-delete.queue_time_us":2475007}
I20240503 14:12:29.134536 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1010) took 2577 ms. Trace:
W20240503 14:12:29.134598 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 19d43cc864064c4c907ec0629391da10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19d43cc864064c4c907ec0629391da10 already in progress: creating tablet
W20240503 14:12:29.135066 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0d4c5648ddc84407a8d8b44f2d63dcee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d4c5648ddc84407a8d8b44f2d63dcee already in progress: creating tablet
I20240503 14:12:29.136695 28604 tablet_bootstrap.cc:492] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:29.136858 28604 ts_tablet_manager.cc:1367] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.212s user 0.006s sys 0.005s
I20240503 14:12:29.138768 28462 tablet_service.cc:1467] Processing CreateTablet for tablet cfe4baf20af348459c229e61fe6d73d5 (DEFAULT_TABLE table=default.Table_af7c6d0b47e24c60b73741ec9e1f6d7b [id=8a61faf53b6a486a921a1b7c55561265]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.138746 29828 rpcz_store.cc:276] 0503 14:12:26.557124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.557615 (+ 491us) service_pool.cc:225] Handling call
0503 14:12:29.134385 (+2576770us) ts_tablet_manager.cc:1120] Deleting tablet d2eb8328e86a4ee5a53647a810729519
0503 14:12:29.134398 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.134517 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2570807}
I20240503 14:12:29.139235 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe4baf20af348459c229e61fe6d73d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.139346 28473 tablet_service.cc:1467] Processing CreateTablet for tablet d0912772bb7446cb9e8f796d24a02225 (DEFAULT_TABLE table=default.Table_7551243835a942e4ac44d0f728b28fe5 [id=6a8e3968b93f4784bc9cc6ac96db8390]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.139544 28468 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6dbc568abd49d896739a36717b993b (DEFAULT_TABLE table=default.Table_9d113544122942a38f6c98e899f8c488 [id=49aafc9bf6624c2b9363fd18ccd242ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.139714 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0912772bb7446cb9e8f796d24a02225. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.140030 28474 tablet_service.cc:1467] Processing CreateTablet for tablet d453860bc12d48efaeafd8bf781859b5 (DEFAULT_TABLE table=default.Table_6a7863d34f3a47f38c253755f3089f6e [id=fadbf65b20be403e8ba3594bc9a95c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.140379 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d453860bc12d48efaeafd8bf781859b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.140340 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2fa8fa721542e9bb814b405b8e6208 (DEFAULT_TABLE table=default.Table_cd02de883e9a43a79775ffc0e5deb283 [id=d8ad2abd71954f60a12fc3e396813bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.140672 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2fa8fa721542e9bb814b405b8e6208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.144408 28471 tablet_service.cc:1467] Processing CreateTablet for tablet 655758ecce9a4d01bf36d4d846436b7b (DEFAULT_TABLE table=default.Table_2300b3e81cff4e629ac39007ac3ba4af [id=32e6719c7fae43d28a5cff49f40d93e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.144614 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 6518f93f7a5c444aa91c9473a5111e5b (DEFAULT_TABLE table=default.Table_a6fe66b4dbee47e9b0b15b491131bfe7 [id=f004db5dd3b745ab8a634efe92efb51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.144860 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655758ecce9a4d01bf36d4d846436b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.145138 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 328bb280f36d4c79b2795c10df2b836b (DEFAULT_TABLE table=default.Table_2bc25daca38a439a9c3275f9cb988224 [id=33e1fdb6383a46f884cae02c5d512f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.145426 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 328bb280f36d4c79b2795c10df2b836b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.139346 28466 tablet_service.cc:1467] Processing CreateTablet for tablet a0f4738ccaad47ab87317ee90efcd46c (DEFAULT_TABLE table=default.Table_70df8ea2254f45aab971123dd7f55a44 [id=24611f8249494aa094924164a4f2c1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.148401 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7fbc0c4cd843efad4fe098e335303e (DEFAULT_TABLE table=default.Table_4cd19da5bf64494e8ff53ded46bc5c4b [id=8adf0b81e1e1448fa0ef8a19c2364eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.148569 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6dbc568abd49d896739a36717b993b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.148742 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f4738ccaad47ab87317ee90efcd46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.148794 29996 raft_consensus.cc:482] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.148850 28459 tablet_service.cc:1467] Processing CreateTablet for tablet eff8442b6c02469eaedb08c51200829b (DEFAULT_TABLE table=default.Table_3d0d6e04014b414aa95434036f1b4516 [id=dd13763c434241bb9352dd30d072c5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.148917 29996 raft_consensus.cc:504] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.149178 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 61da48859c4d4a73bb1b9d7c0ede8106 (DEFAULT_TABLE table=default.Table_c9fcd1ad02784a29858da8e74a9c9866 [id=38c77a1688424f44b3aeeb9384b445bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.149613 29996 leader_election.cc:288] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:29.144408 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 700bc4204db84371bd21f17eefc24521 (DEFAULT_TABLE table=default.Table_1c8ad5b7f0304d83a8cdf65bff9e76e5 [id=f1dc41026a4a421f95b9d6903a2700b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.152438 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6518f93f7a5c444aa91c9473a5111e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.152699 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61da48859c4d4a73bb1b9d7c0ede8106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.153064 28661 ts_tablet_manager.cc:1875] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.155737 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700bc4204db84371bd21f17eefc24521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.156139 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1004) took 2605 ms. Trace:
I20240503 14:12:29.156463 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.156833 29828 rpcz_store.cc:276] 0503 14:12:26.550999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.551086 (+ 87us) service_pool.cc:225] Handling call
0503 14:12:29.155868 (+2604782us) ts_tablet_manager.cc:1120] Deleting tablet c99c15b4d13647e98cae1375b7f13827
0503 14:12:29.155882 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.156107 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2591558}
I20240503 14:12:29.157555 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6696a237c3dd428097e2ebdf24afd63f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.144409 28457 tablet_service.cc:1467] Processing CreateTablet for tablet f87b9edf09814893997ab4578dfe6ee5 (DEFAULT_TABLE table=default.Table_e6e71c3cb9d7443c8354b100cd955068 [id=82dd7ed387914f519c6a8e751c4fa2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.158890 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d348da459634561985c4ac1710c0bb2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.159003 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87b9edf09814893997ab4578dfe6ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.159188 29828 tablet_replica.cc:330] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.159687 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d348da459634561985c4ac1710c0bb2" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.159696 28150 leader_election.cc:302] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.160485 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7fbc0c4cd843efad4fe098e335303e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.164319 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff8442b6c02469eaedb08c51200829b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.165015 29828 raft_consensus.cc:2229] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.165227 29828 raft_consensus.cc:2258] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.165818 29828 ts_tablet_manager.cc:1875] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.165694 28469 tablet_service.cc:1467] Processing CreateTablet for tablet ca74f89636914266a5d9c9d93d3e5d63 (DEFAULT_TABLE table=default.Table_7aa3d349d8e24199ada0764815b8d96e [id=142041e8607541b8bfa097e81d223c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.166128 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca74f89636914266a5d9c9d93d3e5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.168648 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31555fc800142bd849dbd572a194002" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.168903 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf810e84367f420b928a171790849445" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.176252 29828 ts_tablet_manager.cc:1888] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.177170 28661 ts_tablet_manager.cc:1888] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.178869 28661 log.cc:1196] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/ce4846d662a64d51b16fd5447c371e9a
I20240503 14:12:29.179347 28661 ts_tablet_manager.cc:1909] T ce4846d662a64d51b16fd5447c371e9a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.181449 29828 log.cc:1196] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/07354861ee6142d6b1fbe573397d3160
I20240503 14:12:29.181829 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d453860bc12d48efaeafd8bf781859b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.183059 30272 raft_consensus.cc:482] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.183240 30448 raft_consensus.cc:482] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.184057 29828 ts_tablet_manager.cc:1909] T 07354861ee6142d6b1fbe573397d3160 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:29.184181 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 47987.7us
I20240503 14:12:29.184866 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1005) took 2632 ms. Trace:
I20240503 14:12:29.185281 30449 raft_consensus.cc:2781] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.188076 30449 raft_consensus.cc:482] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.188108 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a035a5249f8348b591b27ba3fa7ae43a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.190833 29828 rpcz_store.cc:276] 0503 14:12:26.552657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.552737 (+ 80us) service_pool.cc:225] Handling call
0503 14:12:29.158560 (+2605823us) ts_tablet_manager.cc:1120] Deleting tablet 07354861ee6142d6b1fbe573397d3160
0503 14:12:29.158572 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.173054 (+ 14482us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.176141 (+ 3087us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.184844 (+ 8703us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2594050}
I20240503 14:12:29.191604 29828 tablet_replica.cc:330] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.192133 29828 raft_consensus.cc:2229] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.192675 29828 raft_consensus.cc:2258] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.193558 29828 ts_tablet_manager.cc:1875] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.110016 28614 raft_consensus.cc:348] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.197634 28604 raft_consensus.cc:348] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.197954 28604 raft_consensus.cc:374] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.198040 28604 raft_consensus.cc:729] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.198455 28604 consensus_queue.cc:260] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.199458 28604 ts_tablet_manager.cc:1398] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.063s user 0.003s sys 0.000s
I20240503 14:12:29.199656 28614 raft_consensus.cc:374] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.131376 28290 leader_election.cc:302] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:29.200398 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 69052.7us
I20240503 14:12:29.200803 28604 tablet_bootstrap.cc:492] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:29.200968 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.201478 30431 raft_consensus.cc:2781] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.201579 30431 raft_consensus.cc:482] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.201658 30431 raft_consensus.cc:3037] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.200047 28614 raft_consensus.cc:729] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.202613 28614 consensus_queue.cc:260] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.203361 30272 raft_consensus.cc:504] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.204067 30272 leader_election.cc:288] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.204378 30448 raft_consensus.cc:504] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.204619 28604 tablet_bootstrap.cc:654] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.204977 30448 leader_election.cc:288] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.205236 28213 raft_consensus.cc:2454] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.205389 29996 raft_consensus.cc:482] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.205516 29996 raft_consensus.cc:504] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.197832 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1062) took 1255 ms. Trace:
I20240503 14:12:29.205744 28661 rpcz_store.cc:276] 0503 14:12:27.941998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.955644 (+ 13646us) service_pool.cc:225] Handling call
0503 14:12:29.080043 (+1124399us) ts_tablet_manager.cc:1120] Deleting tablet ce4846d662a64d51b16fd5447c371e9a
0503 14:12:29.080053 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.163387 (+ 83334us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.177068 (+ 13681us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.197813 (+ 20745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124291}
I20240503 14:12:29.205951 28212 raft_consensus.cc:2454] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.206116 29996 leader_election.cc:288] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.206449 28661 tablet_replica.cc:330] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.206635 28661 raft_consensus.cc:2229] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.206799 28661 raft_consensus.cc:2258] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:29.206928 28150 leader_election.cc:341] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.207074 28150 leader_election.cc:302] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:29.207538 28150 leader_election.cc:341] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.207643 28150 leader_election.cc:302] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.208992 28215 raft_consensus.cc:2454] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.209715 30449 raft_consensus.cc:3037] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.214040 28661 ts_tablet_manager.cc:1875] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.215252 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655758ecce9a4d01bf36d4d846436b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.218473 30449 raft_consensus.cc:504] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.221101 30449 leader_election.cc:288] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.205504 30431 raft_consensus.cc:504] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.222023 30431 leader_election.cc:288] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.223460 28614 ts_tablet_manager.cc:1398] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.114s user 0.001s sys 0.002s
I20240503 14:12:29.225311 28427 leader_election.cc:302] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.225618 28427 leader_election.cc:302] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.225833 28427 leader_election.cc:302] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.225891 30446 raft_consensus.cc:2781] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.225991 30446 raft_consensus.cc:482] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.226013 30335 raft_consensus.cc:2781] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.226053 30446 raft_consensus.cc:3037] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.226079 30335 raft_consensus.cc:482] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.226136 30335 raft_consensus.cc:3037] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.226665 30253 consensus_queue.cc:1035] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:29.227078 30451 consensus_queue.cc:1035] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:29.227707 30448 raft_consensus.cc:2726] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.225929 29996 raft_consensus.cc:2781] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.228240 29996 raft_consensus.cc:482] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.228331 29996 raft_consensus.cc:3037] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.228302 30272 raft_consensus.cc:2726] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.228545 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.228916 28212 raft_consensus.cc:2454] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.229511 30446 raft_consensus.cc:504] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:29.230000 28150 leader_election.cc:341] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.230289 30446 leader_election.cc:288] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:29.230561 28150 leader_election.cc:341] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.230926 28150 leader_election.cc:341] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.231060 29996 raft_consensus.cc:504] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.232215 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf810e84367f420b928a171790849445" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:29.232295 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a035a5249f8348b591b27ba3fa7ae43a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.232445 28486 raft_consensus.cc:3037] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.232486 28488 raft_consensus.cc:2379] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:29.233052 29996 leader_election.cc:288] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.233980 30335 raft_consensus.cc:504] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.234572 30335 leader_election.cc:288] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:29.235102 28486 raft_consensus.cc:2454] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:29.232215 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31555fc800142bd849dbd572a194002" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.236151 28493 raft_consensus.cc:2379] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64bc15d65d84ad5a005960a9a84df9c in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:29.236356 28150 leader_election.cc:302] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.236687 30272 raft_consensus.cc:2781] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.236915 30272 raft_consensus.cc:686] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:29.237097 28427 leader_election.cc:302] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.237267 30272 consensus_queue.cc:237] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.238262 29994 raft_consensus.cc:2781] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.238343 29994 raft_consensus.cc:482] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.238412 29994 raft_consensus.cc:3037] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.239662 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31555fc800142bd849dbd572a194002" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.239984 28215 raft_consensus.cc:3037] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.241775 29994 raft_consensus.cc:504] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.242641 29994 leader_election.cc:288] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.243333 28215 raft_consensus.cc:2454] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.243932 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.244693 28213 raft_consensus.cc:3037] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.248303 29994 raft_consensus.cc:482] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.248456 29994 raft_consensus.cc:504] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.249184 29994 leader_election.cc:288] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:29.252550 28290 leader_election.cc:341] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 13c7a51c68ef478e9bdc5b3e3dc3d5df
I20240503 14:12:29.255798 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31555fc800142bd849dbd572a194002" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.255837 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.244366 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a035a5249f8348b591b27ba3fa7ae43a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.256197 28348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf810e84367f420b928a171790849445" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.256353 28214 raft_consensus.cc:3037] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.256417 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a035a5249f8348b591b27ba3fa7ae43a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.256616 28346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.256827 28345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31555fc800142bd849dbd572a194002" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.257083 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.257644 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a035a5249f8348b591b27ba3fa7ae43a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.257915 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a035a5249f8348b591b27ba3fa7ae43a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.258373 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31555fc800142bd849dbd572a194002" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:29.258561 28150 leader_election.cc:341] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.258713 28150 leader_election.cc:302] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.258880 28614 tablet_bootstrap.cc:492] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:29.259224 28150 leader_election.cc:341] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.259538 28150 leader_election.cc:341] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.259685 28150 leader_election.cc:302] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.260360 28661 ts_tablet_manager.cc:1888] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.260489 28661 log.cc:1196] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/e0175b38c1e0464a9a6208cfaf68ca0d
I20240503 14:12:29.260831 28661 ts_tablet_manager.cc:1909] T e0175b38c1e0464a9a6208cfaf68ca0d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.262702 28213 raft_consensus.cc:2454] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.263043 28214 raft_consensus.cc:2454] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.263238 30448 raft_consensus.cc:2726] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.263281 30449 raft_consensus.cc:482] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.263387 30449 raft_consensus.cc:504] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.264003 30449 leader_election.cc:288] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.264333 28427 leader_election.cc:302] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.264845 28427 leader_election.cc:302] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.264914 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167a926fa61747e2b4fcb52ba4eeea5b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.265070 28427 leader_election.cc:302] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.265398 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167a926fa61747e2b4fcb52ba4eeea5b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.265960 30446 raft_consensus.cc:2781] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.266090 30446 raft_consensus.cc:686] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.266162 29996 raft_consensus.cc:2781] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.266284 29996 raft_consensus.cc:686] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.266305 30446 consensus_queue.cc:237] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.266534 29996 consensus_queue.cc:237] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.266875 29994 raft_consensus.cc:2781] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.267000 29994 raft_consensus.cc:686] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.267187 29994 consensus_queue.cc:237] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.267896 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de9e159d2ba47039dd6bd207a9cddeb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.268126 28215 raft_consensus.cc:2454] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.268833 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.269062 28213 raft_consensus.cc:3037] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.273391 28213 raft_consensus.cc:2454] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.274900 28427 leader_election.cc:302] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:29.275524 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet addc2a6195034b0a927fdee7a2955122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet addc2a6195034b0a927fdee7a2955122 already in progress: creating tablet
W20240503 14:12:29.281970 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 77511682d71a40f5933edcb6a6523649 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77511682d71a40f5933edcb6a6523649 already in progress: creating tablet
I20240503 14:12:29.282465 29996 raft_consensus.cc:2781] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.282680 29996 raft_consensus.cc:482] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.282805 29996 raft_consensus.cc:3037] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.273650 30272 raft_consensus.cc:2726] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.284475 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1063) took 1319 ms. Trace:
I20240503 14:12:29.279320 30266 raft_consensus.cc:482] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.284795 28661 rpcz_store.cc:276] 0503 14:12:27.942071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.955770 (+ 13699us) service_pool.cc:225] Handling call
0503 14:12:29.206390 (+1250620us) ts_tablet_manager.cc:1120] Deleting tablet e0175b38c1e0464a9a6208cfaf68ca0d
0503 14:12:29.206402 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.244699 (+ 38297us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.260207 (+ 15508us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.261624 (+ 1417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1250517}
I20240503 14:12:29.285044 30266 raft_consensus.cc:504] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.285465 28661 tablet_replica.cc:330] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.285682 28661 raft_consensus.cc:2229] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.285869 28661 raft_consensus.cc:2258] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.286279 30266 leader_election.cc:288] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.287530 28661 ts_tablet_manager.cc:1875] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.290508 28427 leader_election.cc:302] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.292492 29994 raft_consensus.cc:2781] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.293292 28614 tablet_bootstrap.cc:654] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.294099 29996 raft_consensus.cc:504] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.294728 29994 raft_consensus.cc:686] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.294924 29996 leader_election.cc:288] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.295241 29994 consensus_queue.cc:237] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.296134 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de9e159d2ba47039dd6bd207a9cddeb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.296542 28213 raft_consensus.cc:3037] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.298982 28604 tablet_bootstrap.cc:492] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:29.299221 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 11ac912a3bd64cdc9082596e09c72e13 (table default.Table_78f62674ace541fabf9047f21a7cadbc [id=aed2a71b61da4d3981c77404ba8dffaa]) successfully deleted
W20240503 14:12:29.299688 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d2eb8328e86a4ee5a53647a810729519 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2eb8328e86a4ee5a53647a810729519 already in progress: creating tablet
I20240503 14:12:29.299813 28604 ts_tablet_manager.cc:1367] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
W20240503 14:12:29.281275 28150 leader_election.cc:341] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.301095 28150 leader_election.cc:341] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.301236 28150 leader_election.cc:302] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.301587 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c7a51c68ef478e9bdc5b3e3dc3d5df" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:29.301657 30431 raft_consensus.cc:482] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.301805 28493 raft_consensus.cc:3037] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.301806 30431 raft_consensus.cc:504] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.302439 30431 leader_election.cc:288] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.302930 30272 raft_consensus.cc:2726] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.304279 30253 raft_consensus.cc:482] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.304445 30253 raft_consensus.cc:504] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.305200 30253 leader_election.cc:288] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.305450 28493 raft_consensus.cc:2454] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:29.314139 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.314498 28207 raft_consensus.cc:2454] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.315059 28604 raft_consensus.cc:348] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.315832 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c79ffee3954b5c870f4a8aa6379517" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.317080 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c79ffee3954b5c870f4a8aa6379517" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.318131 28604 raft_consensus.cc:374] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.318396 28604 raft_consensus.cc:729] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.318795 28604 consensus_queue.cc:260] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.320124 28604 ts_tablet_manager.cc:1398] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 14:12:29.321008 28604 tablet_bootstrap.cc:492] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:29.323282 28604 tablet_bootstrap.cc:654] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.324982 28661 ts_tablet_manager.cc:1888] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.328034 28661 log.cc:1196] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/920b41e3b21347b393497de4d4e87469
I20240503 14:12:29.328541 28661 ts_tablet_manager.cc:1909] T 920b41e3b21347b393497de4d4e87469 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.329933 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.330641 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de9e159d2ba47039dd6bd207a9cddeb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:29.330916 28427 leader_election.cc:341] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.331636 28427 leader_election.cc:341] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.331972 28427 leader_election.cc:341] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.331926 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de9e159d2ba47039dd6bd207a9cddeb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:29.332259 28427 leader_election.cc:341] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.332634 28427 consensus_peers.cc:485] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.332649 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:29.332782 28427 leader_election.cc:341] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.333498 28290 leader_election.cc:341] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 13c7a51c68ef478e9bdc5b3e3dc3d5df
W20240503 14:12:29.333771 28290 leader_election.cc:341] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 920b41e3b21347b393497de4d4e87469
I20240503 14:12:29.334034 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1064) took 1391 ms. Trace:
I20240503 14:12:29.334127 28661 rpcz_store.cc:276] 0503 14:12:27.942146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.955889 (+ 13743us) service_pool.cc:225] Handling call
0503 14:12:29.285401 (+1329512us) ts_tablet_manager.cc:1120] Deleting tablet 920b41e3b21347b393497de4d4e87469
0503 14:12:29.285415 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.311981 (+ 26566us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.324857 (+ 12876us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.334009 (+ 9152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329409}
W20240503 14:12:29.334306 28150 leader_election.cc:341] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.334528 28150 leader_election.cc:341] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.334578 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1059) took 1392 ms. Trace:
I20240503 14:12:29.334614 28150 leader_election.cc:302] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.334656 28661 rpcz_store.cc:276] 0503 14:12:27.941730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.956578 (+ 14848us) service_pool.cc:225] Handling call
0503 14:12:29.334443 (+1377865us) ts_tablet_manager.cc:1120] Deleting tablet 7549b2ca31b04247bf3a2202e7585817
0503 14:12:29.334455 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.334558 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377684}
I20240503 14:12:29.334888 30272 raft_consensus.cc:2726] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:29.334798 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c99c15b4d13647e98cae1375b7f13827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c99c15b4d13647e98cae1375b7f13827 already in progress: creating tablet
I20240503 14:12:29.335268 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 07354861ee6142d6b1fbe573397d3160 (table default.Table_7fbc01371cd145eeb7b51110d14f6456 [id=2cc7a41775044f4a993bd36fc229d152]) successfully deleted
I20240503 14:12:29.335923 28213 raft_consensus.cc:2454] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.335007 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1069) took 1309 ms. Trace:
I20240503 14:12:29.336205 28661 rpcz_store.cc:276] 0503 14:12:28.025768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.025843 (+ 75us) service_pool.cc:225] Handling call
0503 14:12:29.334894 (+1309051us) ts_tablet_manager.cc:1120] Deleting tablet 91eb4433685b4119a170054a0a777e05
0503 14:12:29.334904 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.334989 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308802}
I20240503 14:12:29.337954 28197 tablet_service.cc:1467] Processing CreateTablet for tablet cc815484ff24435fab5d994e4e564668 (DEFAULT_TABLE table=default.Table_b71e58c4b31248888eecec2d163d0181 [id=e4a9eb2d39d34430bde5d4aae9dccadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.338516 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc815484ff24435fab5d994e4e564668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.341009 28186 tablet_service.cc:1467] Processing CreateTablet for tablet d0455a7726fb4ec18a42e1ef301bdc34 (DEFAULT_TABLE table=default.Table_0e114c978c714945b50f17d008fb60d4 [id=2f530d39a4334126b6476374aa834e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.341545 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0455a7726fb4ec18a42e1ef301bdc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.342589 28180 tablet_service.cc:1467] Processing CreateTablet for tablet cc9eff03c4f142ad888420043a18d7fc (DEFAULT_TABLE table=default.Table_33a3711d0f824b1499cb143a33f44014 [id=8aa44daef41d466eab5e603fab038493]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.342990 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9eff03c4f142ad888420043a18d7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.343721 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 1133ad5c30214f04808e85c918185923 (DEFAULT_TABLE table=default.Table_77a20b54283b43be95a5e60a0a9dbd05 [id=eb0876813d4140b8a640fe0d08ab01d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.341130 28661 tablet_replica.cc:330] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.344127 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1133ad5c30214f04808e85c918185923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.344111 28661 raft_consensus.cc:2229] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.344322 28661 raft_consensus.cc:2258] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.344969 28661 ts_tablet_manager.cc:1875] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.345175 28187 tablet_service.cc:1467] Processing CreateTablet for tablet f0c85614221c4e34811d02ecd2f7ce5d (DEFAULT_TABLE table=default.Table_346e6ce2bc2b4524a52ec6ac63cbe9d8 [id=5e49cdc6f80642deb0d21f48dd001788]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.345427 28614 tablet_bootstrap.cc:492] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.345546 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c85614221c4e34811d02ecd2f7ce5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.345577 28614 ts_tablet_manager.cc:1367] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20240503 14:12:29.346278 28196 tablet_service.cc:1467] Processing CreateTablet for tablet f53643c4a69648989a4bd0c30c5dd24f (DEFAULT_TABLE table=default.Table_2fe7f4a5bc6f467e890e502b73d2545c [id=8631dcb5087e4ded87da93fc930c2031]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.346359 28614 raft_consensus.cc:348] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.346585 28614 raft_consensus.cc:374] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.346634 28614 raft_consensus.cc:729] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.346679 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53643c4a69648989a4bd0c30c5dd24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.346844 28614 consensus_queue.cc:260] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.347549 28614 ts_tablet_manager.cc:1398] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:29.347880 28614 tablet_bootstrap.cc:492] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.347986 28604 tablet_bootstrap.cc:492] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:29.348145 28604 ts_tablet_manager.cc:1367] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240503 14:12:29.349074 28604 raft_consensus.cc:348] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.349434 28614 tablet_bootstrap.cc:654] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.349736 28604 raft_consensus.cc:374] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.349874 28604 raft_consensus.cc:729] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.350173 28604 consensus_queue.cc:260] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.350925 28604 ts_tablet_manager.cc:1398] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:29.351423 28604 tablet_bootstrap.cc:492] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:29.352363 30272 raft_consensus.cc:482] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.352514 30272 raft_consensus.cc:504] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.353196 30272 leader_election.cc:288] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.353390 29828 ts_tablet_manager.cc:1888] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.353516 29828 log.cc:1196] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/57ab077988364bd687e54f90c3c265d5
I20240503 14:12:29.353868 29828 ts_tablet_manager.cc:1909] T 57ab077988364bd687e54f90c3c265d5 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.354207 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646f83185d5241268d4126d10561e937" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.354460 28604 tablet_bootstrap.cc:654] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:29.355131 28427 leader_election.cc:341] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.355538 28427 leader_election.cc:341] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.355598 28150 leader_election.cc:341] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.355847 28427 leader_election.cc:341] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.355856 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646f83185d5241268d4126d10561e937" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.356158 28493 raft_consensus.cc:2454] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:29.356498 28330 tablet_service.cc:1467] Processing CreateTablet for tablet f0c85614221c4e34811d02ecd2f7ce5d (DEFAULT_TABLE table=default.Table_346e6ce2bc2b4524a52ec6ac63cbe9d8 [id=5e49cdc6f80642deb0d21f48dd001788]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.356971 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c85614221c4e34811d02ecd2f7ce5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:29.357411 28427 leader_election.cc:341] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.358788 28427 leader_election.cc:341] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.358989 28427 leader_election.cc:341] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.359915 28150 leader_election.cc:302] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:29.360814 28600 tablet_bootstrap.cc:492] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.360977 28600 ts_tablet_manager.cc:1367] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.347s user 0.068s sys 0.113s
I20240503 14:12:29.361745 28600 raft_consensus.cc:348] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.361953 28600 raft_consensus.cc:374] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.362007 28600 raft_consensus.cc:729] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.362207 28600 consensus_queue.cc:260] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.362720 28600 ts_tablet_manager.cc:1398] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:29.363023 28600 tablet_bootstrap.cc:492] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:29.364255 28321 tablet_service.cc:1467] Processing CreateTablet for tablet d0455a7726fb4ec18a42e1ef301bdc34 (DEFAULT_TABLE table=default.Table_0e114c978c714945b50f17d008fb60d4 [id=2f530d39a4334126b6476374aa834e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.364459 28600 tablet_bootstrap.cc:654] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.364481 28318 tablet_service.cc:1467] Processing CreateTablet for tablet cc9eff03c4f142ad888420043a18d7fc (DEFAULT_TABLE table=default.Table_33a3711d0f824b1499cb143a33f44014 [id=8aa44daef41d466eab5e603fab038493]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.364714 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0455a7726fb4ec18a42e1ef301bdc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.364920 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9eff03c4f142ad888420043a18d7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.367851 28469 tablet_service.cc:1467] Processing CreateTablet for tablet f53643c4a69648989a4bd0c30c5dd24f (DEFAULT_TABLE table=default.Table_2fe7f4a5bc6f467e890e502b73d2545c [id=8631dcb5087e4ded87da93fc930c2031]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.368328 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53643c4a69648989a4bd0c30c5dd24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.367996 28330 tablet_service.cc:1467] Processing CreateTablet for tablet cc815484ff24435fab5d994e4e564668 (DEFAULT_TABLE table=default.Table_b71e58c4b31248888eecec2d163d0181 [id=e4a9eb2d39d34430bde5d4aae9dccadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.368659 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc815484ff24435fab5d994e4e564668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.372684 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 1133ad5c30214f04808e85c918185923 (DEFAULT_TABLE table=default.Table_77a20b54283b43be95a5e60a0a9dbd05 [id=eb0876813d4140b8a640fe0d08ab01d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.373134 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1133ad5c30214f04808e85c918185923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.375586 28469 tablet_service.cc:1467] Processing CreateTablet for tablet f0c85614221c4e34811d02ecd2f7ce5d (DEFAULT_TABLE table=default.Table_346e6ce2bc2b4524a52ec6ac63cbe9d8 [id=5e49cdc6f80642deb0d21f48dd001788]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.376000 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c85614221c4e34811d02ecd2f7ce5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.379367 28604 tablet_bootstrap.cc:492] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:29.379545 28604 ts_tablet_manager.cc:1367] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240503 14:12:29.380430 28604 raft_consensus.cc:348] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.380724 28604 raft_consensus.cc:374] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.380811 28604 raft_consensus.cc:729] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.381141 28604 consensus_queue.cc:260] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.381803 28604 ts_tablet_manager.cc:1398] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:29.382171 28604 tablet_bootstrap.cc:492] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:29.382601 28461 tablet_service.cc:1467] Processing CreateTablet for tablet cc815484ff24435fab5d994e4e564668 (DEFAULT_TABLE table=default.Table_b71e58c4b31248888eecec2d163d0181 [id=e4a9eb2d39d34430bde5d4aae9dccadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.382982 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc815484ff24435fab5d994e4e564668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.384049 28604 tablet_bootstrap.cc:654] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.385564 28661 ts_tablet_manager.cc:1888] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.386565 28661 log.cc:1196] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/02d45788ae644a8581acca54cc55dd3f
I20240503 14:12:29.387027 28661 ts_tablet_manager.cc:1909] T 02d45788ae644a8581acca54cc55dd3f P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.391095 28461 tablet_service.cc:1467] Processing CreateTablet for tablet d0455a7726fb4ec18a42e1ef301bdc34 (DEFAULT_TABLE table=default.Table_0e114c978c714945b50f17d008fb60d4 [id=2f530d39a4334126b6476374aa834e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.391573 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0455a7726fb4ec18a42e1ef301bdc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.391870 28661 tablet_replica.cc:330] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.392165 28661 raft_consensus.cc:2229] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:29.392293 30272 raft_consensus.cc:2781] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.392392 30272 raft_consensus.cc:482] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.392446 30272 raft_consensus.cc:3037] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.392624 28661 pending_rounds.cc:62] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:29.392676 28661 pending_rounds.cc:68] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597763043164160 op_type: NO_OP noop_request { }
I20240503 14:12:29.392863 28661 raft_consensus.cc:2866] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:29.392949 28661 raft_consensus.cc:2258] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.393571 28661 ts_tablet_manager.cc:1875] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.395098 30272 raft_consensus.cc:504] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.395803 30272 leader_election.cc:288] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.396318 28290 leader_election.cc:302] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:29.396641 30431 raft_consensus.cc:2781] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.396808 30431 raft_consensus.cc:686] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:29.397079 30431 consensus_queue.cc:237] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.397665 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646f83185d5241268d4126d10561e937" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.398226 28466 tablet_service.cc:1467] Processing CreateTablet for tablet cc9eff03c4f142ad888420043a18d7fc (DEFAULT_TABLE table=default.Table_33a3711d0f824b1499cb143a33f44014 [id=8aa44daef41d466eab5e603fab038493]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.398938 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646f83185d5241268d4126d10561e937" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:29.399166 28493 raft_consensus.cc:3037] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.399384 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 1133ad5c30214f04808e85c918185923 (DEFAULT_TABLE table=default.Table_77a20b54283b43be95a5e60a0a9dbd05 [id=eb0876813d4140b8a640fe0d08ab01d2]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:29.400297 28150 leader_election.cc:341] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.354207 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1006) took 2801 ms. Trace:
I20240503 14:12:29.401408 29828 rpcz_store.cc:276] 0503 14:12:26.552936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.552989 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:29.191542 (+2638553us) ts_tablet_manager.cc:1120] Deleting tablet 57ab077988364bd687e54f90c3c265d5
0503 14:12:29.191556 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.200026 (+ 8470us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.353254 (+153228us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.354187 (+ 933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2627004}
I20240503 14:12:29.401749 29828 tablet_replica.cc:330] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.401947 29828 raft_consensus.cc:2229] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.402122 29828 raft_consensus.cc:2258] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.402141 28493 raft_consensus.cc:2454] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 1.
I20240503 14:12:29.403048 28150 leader_election.cc:302] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:29.403331 30272 raft_consensus.cc:2781] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.403779 30272 raft_consensus.cc:686] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Becoming Leader. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Running, Role: LEADER
I20240503 14:12:29.404208 30272 consensus_queue.cc:237] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.406047 28427 leader_election.cc:302] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:29.406841 30266 raft_consensus.cc:482] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.406970 30266 raft_consensus.cc:504] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.407200 30335 raft_consensus.cc:2781] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.407287 30335 raft_consensus.cc:482] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.407347 30335 raft_consensus.cc:3037] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.407567 30266 leader_election.cc:288] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.408545 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdc868ddf0a490292eae78ebc8c8ec6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:29.409595 28150 leader_election.cc:341] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.409683 28290 leader_election.cc:341] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 920b41e3b21347b393497de4d4e87469
I20240503 14:12:29.409826 28290 leader_election.cc:302] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.409830 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdc868ddf0a490292eae78ebc8c8ec6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.410027 29828 ts_tablet_manager.cc:1875] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.410086 30335 raft_consensus.cc:504] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.410182 30263 raft_consensus.cc:2726] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.410651 30335 leader_election.cc:288] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.411491 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9eff03c4f142ad888420043a18d7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.412003 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1133ad5c30214f04808e85c918185923. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:29.412832 28150 leader_election.cc:341] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.412938 28150 leader_election.cc:302] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.413607 30266 raft_consensus.cc:2726] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.418242 28661 ts_tablet_manager.cc:1888] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.419764 30335 raft_consensus.cc:482] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.420192 30335 raft_consensus.cc:504] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.421416 30335 leader_election.cc:288] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:29.421695 28661 log.cc:1196] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/11e14e8b5f4046029af4673a277112d9
I20240503 14:12:29.424991 28661 ts_tablet_manager.cc:1909] T 11e14e8b5f4046029af4673a277112d9 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.423951 28427 leader_election.cc:302] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:29.425441 30335 raft_consensus.cc:2781] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.426010 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet c9325d5142af4b018354bf38370b61e0 (table default.Table_2ce3b6f2b77b4607856fc2a64ead5bb5 [id=db393a7e2a83466385ffc0af3d2789c4]) successfully deleted
I20240503 14:12:29.426175 30335 raft_consensus.cc:686] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.426566 30335 consensus_queue.cc:237] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.428669 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.428910 28207 raft_consensus.cc:3037] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.431067 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49445f0f1dd1451c9ba066e61273f2f1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.431300 28212 raft_consensus.cc:2454] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.431805 28207 raft_consensus.cc:2454] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.433797 29828 ts_tablet_manager.cc:1888] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.434021 29828 log.cc:1196] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/20b176a071ba4f3fbff2a64d6e46027a
I20240503 14:12:29.433351 28427 leader_election.cc:302] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.434511 29996 raft_consensus.cc:2781] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.434546 29828 ts_tablet_manager.cc:1909] T 20b176a071ba4f3fbff2a64d6e46027a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.434553 28427 leader_election.cc:302] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.434612 29996 raft_consensus.cc:482] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.434679 29996 raft_consensus.cc:3037] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:29.435520 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet b97da778da214bb6bc147c5ada6c7603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b97da778da214bb6bc147c5ada6c7603 already in progress: creating tablet
I20240503 14:12:29.435634 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1008) took 2878 ms. Trace:
I20240503 14:12:29.435722 30446 raft_consensus.cc:2781] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.435838 30446 raft_consensus.cc:686] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.436158 30446 consensus_queue.cc:237] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.436432 29828 rpcz_store.cc:276] 0503 14:12:26.556911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.557504 (+ 593us) service_pool.cc:225] Handling call
0503 14:12:29.401683 (+2844179us) ts_tablet_manager.cc:1120] Deleting tablet 20b176a071ba4f3fbff2a64d6e46027a
0503 14:12:29.401693 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.417651 (+ 15958us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.433639 (+ 15988us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.435603 (+ 1964us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2795659}
I20240503 14:12:29.437247 29996 raft_consensus.cc:504] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.437808 29996 leader_election.cc:288] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:29.437371 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1009) took 2880 ms. Trace:
I20240503 14:12:29.438021 29828 rpcz_store.cc:276] 0503 14:12:26.557005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.557531 (+ 526us) service_pool.cc:225] Handling call
0503 14:12:29.437251 (+2879720us) ts_tablet_manager.cc:1120] Deleting tablet 2221218847554cbd9d3242a9a6431e46
0503 14:12:29.437261 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.437351 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2824802}
I20240503 14:12:29.439046 28661 tablet_replica.cc:330] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:29.440500 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f91ddbc250c44fbfbf4ffe83433d8dd8 already in progress: creating tablet
I20240503 14:12:29.441596 28661 raft_consensus.cc:2229] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.442149 28344 raft_consensus.cc:3037] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.442294 28661 raft_consensus.cc:2258] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.444610 28614 tablet_bootstrap.cc:492] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.444795 28614 ts_tablet_manager.cc:1367] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20240503 14:12:29.445729 28661 ts_tablet_manager.cc:1875] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.445906 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1011) took 2838 ms. Trace:
I20240503 14:12:29.445957 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet dbcc6cef7b7d464a9fbbae61caf0437d (table default.Table_7c4e797497e74bbe98ff720b9ffd05ff [id=ee4e5312119d414aa8f8d01c28e0b5fb]) successfully deleted
I20240503 14:12:29.446060 29828 rpcz_store.cc:276] 0503 14:12:26.607706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.608053 (+ 347us) service_pool.cc:225] Handling call
0503 14:12:29.445767 (+2837714us) ts_tablet_manager.cc:1120] Deleting tablet 7b7ca19dd15f48a68ace7fefc7ae660f
0503 14:12:29.445779 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.445887 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2829137}
I20240503 14:12:29.446650 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1013) took 2817 ms. Trace:
I20240503 14:12:29.446789 29828 rpcz_store.cc:276] 0503 14:12:26.629349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.629406 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:29.446526 (+2817120us) ts_tablet_manager.cc:1120] Deleting tablet 7d348da459634561985c4ac1710c0bb2
0503 14:12:29.446537 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.446634 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2816892}
I20240503 14:12:29.447110 29828 tablet_replica.cc:330] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.447351 29828 raft_consensus.cc:2229] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.447638 29828 raft_consensus.cc:2258] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.448318 29828 ts_tablet_manager.cc:1875] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.449399 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49445f0f1dd1451c9ba066e61273f2f1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.449635 28213 raft_consensus.cc:3037] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.452656 28213 raft_consensus.cc:2454] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.459435 28427 leader_election.cc:302] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.459842 30263 raft_consensus.cc:482] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.460002 30263 raft_consensus.cc:504] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.460557 30446 raft_consensus.cc:2781] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.460696 30446 raft_consensus.cc:686] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.460706 30263 leader_election.cc:288] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.460937 30446 consensus_queue.cc:237] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.462651 28661 ts_tablet_manager.cc:1888] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.463246 29828 ts_tablet_manager.cc:1888] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 14:12:29.463457 29828 log.cc:1196] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/5b90e9563b4c4cef8ee7964eaa96e285
I20240503 14:12:29.463836 28661 log.cc:1196] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/9ac2833a0bb045dcb2824468d936ffe4
I20240503 14:12:29.463882 29828 ts_tablet_manager.cc:1909] T 5b90e9563b4c4cef8ee7964eaa96e285 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.464509 28661 ts_tablet_manager.cc:1909] T 9ac2833a0bb045dcb2824468d936ffe4 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.464834 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1012) took 2835 ms. Trace:
I20240503 14:12:29.465561 28661 tablet_replica.cc:330] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.467772 28661 raft_consensus.cc:2229] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.470276 28661 raft_consensus.cc:2258] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.473030 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 3d54cc5c03034a6685c1da34f9bd65ac (table default.Table_8f505fa519514278ae7489b9ff7bbaf2 [id=f58e9b32e96948cfab2760b64eb18b68]) successfully deleted
I20240503 14:12:29.474120 29828 rpcz_store.cc:276] 0503 14:12:26.629136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.629746 (+ 610us) service_pool.cc:225] Handling call
0503 14:12:29.447066 (+2817320us) ts_tablet_manager.cc:1120] Deleting tablet 5b90e9563b4c4cef8ee7964eaa96e285
0503 14:12:29.447075 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.451566 (+ 4491us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.463110 (+ 11544us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.464805 (+ 1695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2817152}
I20240503 14:12:29.474658 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1014) took 2716 ms. Trace:
I20240503 14:12:29.474787 29828 rpcz_store.cc:276] 0503 14:12:26.758138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.758196 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:29.474507 (+2716311us) ts_tablet_manager.cc:1120] Deleting tablet 728ba3ac1cb645168481b84b5b548140
0503 14:12:29.474518 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.474639 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2716117}
I20240503 14:12:29.475034 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49445f0f1dd1451c9ba066e61273f2f1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.475169 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1017) took 2695 ms. Trace:
I20240503 14:12:29.475239 29828 rpcz_store.cc:276] 0503 14:12:26.779257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.788277 (+ 9020us) service_pool.cc:225] Handling call
0503 14:12:29.475055 (+2686778us) ts_tablet_manager.cc:1120] Deleting tablet bab214b7ec58425d8cd1180ee16b273d
0503 14:12:29.475063 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.475152 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2686555}
I20240503 14:12:29.475528 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1018) took 2682 ms. Trace:
I20240503 14:12:29.475600 29828 rpcz_store.cc:276] 0503 14:12:26.792939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.796413 (+ 3474us) service_pool.cc:225] Handling call
0503 14:12:29.475435 (+2679022us) ts_tablet_manager.cc:1120] Deleting tablet c725628f72884086b523b56ecf876069
0503 14:12:29.475443 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.475514 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2678800}
I20240503 14:12:29.475826 29828 tablet_replica.cc:330] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.476024 29828 raft_consensus.cc:2229] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.476222 29828 raft_consensus.cc:2258] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:29.476058 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5500da0e5658492b840dd29273ab265f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5500da0e5658492b840dd29273ab265f already in progress: creating tablet
I20240503 14:12:29.477164 29828 ts_tablet_manager.cc:1875] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:29.477200 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 5685a6b3ea6943f8a72d74de87ac2166 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5685a6b3ea6943f8a72d74de87ac2166 already in progress: creating tablet
I20240503 14:12:29.469085 28614 raft_consensus.cc:348] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.473428 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.478729 28614 raft_consensus.cc:374] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.473970 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49445f0f1dd1451c9ba066e61273f2f1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.478847 28614 raft_consensus.cc:729] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.479647 28614 consensus_queue.cc:260] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.480255 30263 raft_consensus.cc:482] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.480388 30263 raft_consensus.cc:504] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.481086 30263 leader_election.cc:288] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.481654 30272 raft_consensus.cc:482] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.481806 30272 raft_consensus.cc:504] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.482421 30272 leader_election.cc:288] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.483163 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59945d011b64cba87c854e2ffde8226" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:29.483991 28150 leader_election.cc:341] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.484715 28614 ts_tablet_manager.cc:1398] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240503 14:12:29.487452 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59945d011b64cba87c854e2ffde8226" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.488268 28150 leader_election.cc:341] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.488425 28150 leader_election.cc:302] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.488894 30272 raft_consensus.cc:2726] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.489485 28614 tablet_bootstrap.cc:492] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.491178 28614 tablet_bootstrap.cc:654] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.493610 30335 consensus_queue.cc:1035] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:29.494014 28427 leader_election.cc:341] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.494467 28427 leader_election.cc:341] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.494741 28427 leader_election.cc:341] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.495345 29828 ts_tablet_manager.cc:1888] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.495476 29828 log.cc:1196] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/3d54cc5c03034a6685c1da34f9bd65ac
I20240503 14:12:29.495846 29828 ts_tablet_manager.cc:1909] T 3d54cc5c03034a6685c1da34f9bd65ac P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.496714 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1027) took 2583 ms. Trace:
I20240503 14:12:29.503268 29828 rpcz_store.cc:276] 0503 14:12:26.912835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.913131 (+ 296us) service_pool.cc:225] Handling call
0503 14:12:29.475783 (+2562652us) ts_tablet_manager.cc:1120] Deleting tablet 3d54cc5c03034a6685c1da34f9bd65ac
0503 14:12:29.475790 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.486798 (+ 11008us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.495213 (+ 8415us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.496684 (+ 1471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2562404}
I20240503 14:12:29.477761 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 09f1f5216cc940888803ce8f6acd2a47
I20240503 14:12:29.504276 29828 tablet_replica.cc:330] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.504473 29828 raft_consensus.cc:2229] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.504498 28661 ts_tablet_manager.cc:1875] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:29.504487 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 3dfd6d40a57e4bea973874ff12c0ea3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dfd6d40a57e4bea973874ff12c0ea3a already in progress: creating tablet
I20240503 14:12:29.504657 29828 raft_consensus.cc:2258] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.504705 28025 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1228) took 1526 ms. Trace:
I20240503 14:12:29.504805 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a3d1314841574fc8bcf3ccb16755e421 (table default.Table_ddd5f026bb204de99edef3cc54bd8347 [id=e5c443cd73d340e99806c89d5b8e659f]) successfully deleted
I20240503 14:12:29.504825 28025 rpcz_store.cc:276] 0503 14:12:27.978104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.772559 (+794455us) service_pool.cc:225] Handling call
0503 14:12:28.772983 (+ 424us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.773138 (+ 155us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:28.773212 (+ 74us) catalog_manager.cc:2758] Locking tablets
0503 14:12:28.783243 (+ 10031us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:29.477534 (+694291us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:29.477664 (+ 130us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:29.477725 (+ 61us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:29.504629 (+ 26904us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.807556 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.807701 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.807708 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:28.807710 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.807838 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.807850 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.807856 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.807963 (+ 107us) tablet.cc:699] Row locks acquired
0503 14:12:28.807966 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.808009 (+ 43us) write_op.cc:270] Start()
0503 14:12:28.808046 (+ 37us) write_op.cc:276] Timestamp: P: 1714745548807981 usec, L: 0
0503 14:12:28.808050 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.808317 (+ 267us) log.cc:842] Serialized 762 byte log entry
0503 14:12:29.474018 (+665701us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.476385 (+ 2367us) write_op.cc:301] APPLY: starting
0503 14:12:29.476406 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.476460 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.476489 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.476572 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.476614 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:29.476622 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:29.476730 (+ 108us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.476808 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.476834 (+ 26us) write_op.cc:454] Released schema lock
0503 14:12:29.476842 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.478473 (+ 1631us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.478478 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2306,"apply.run_cpu_time_us":567,"apply.run_wall_time_us":2137,"num_ops":2,"prepare.queue_time_us":23626,"prepare.run_cpu_time_us":855,"prepare.run_wall_time_us":1495,"replication_time_us":665942,"thread_start_us":2196,"threads_started":1}]]}
I20240503 14:12:29.506106 29828 ts_tablet_manager.cc:1875] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.506166 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet ce4846d662a64d51b16fd5447c371e9a (table default.Table_4fdbe31ba57a42b993d60fa19081c044 [id=ea4ef9402edd4db9820a6a318bfec37d]) successfully deleted
I20240503 14:12:29.506644 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet e0175b38c1e0464a9a6208cfaf68ca0d (table default.Table_7fdf02629122402c9cbf8053f89eced3 [id=5201e94e1d164116b74d4329e5991eca]) successfully deleted
I20240503 14:12:29.506863 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 920b41e3b21347b393497de4d4e87469 (table default.Table_28c16c4a952d45c0b34fd1d4864ac748 [id=2fd597e69c1041a2984ca535058f6fab]) successfully deleted
W20240503 14:12:29.507086 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 7549b2ca31b04247bf3a2202e7585817 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7549b2ca31b04247bf3a2202e7585817 already in progress: creating tablet
W20240503 14:12:29.507277 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 91eb4433685b4119a170054a0a777e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91eb4433685b4119a170054a0a777e05 already in progress: creating tablet
W20240503 14:12:29.507476 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e220336186f648419b186feada6f0124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e220336186f648419b186feada6f0124 already in progress: creating tablet
I20240503 14:12:29.507863 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 02d45788ae644a8581acca54cc55dd3f (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
I20240503 14:12:29.478171 28028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1229) took 1499 ms. Trace:
I20240503 14:12:29.508340 28028 rpcz_store.cc:276] 0503 14:12:27.978198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.774236 (+796038us) service_pool.cc:225] Handling call
0503 14:12:28.775191 (+ 955us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:28.775700 (+ 509us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:29.477880 (+702180us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:29.478003 (+ 123us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:29.478111 (+ 108us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.779105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.779236 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.779243 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:28.779246 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:28.779331 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.779351 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.779356 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.779429 (+ 73us) tablet.cc:699] Row locks acquired
0503 14:12:28.779432 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.779469 (+ 37us) write_op.cc:270] Start()
0503 14:12:28.779571 (+ 102us) write_op.cc:276] Timestamp: P: 1714745548779445 usec, L: 0
0503 14:12:28.779576 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.779787 (+ 211us) log.cc:842] Serialized 433 byte log entry
0503 14:12:29.415690 (+635903us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.469561 (+ 53871us) write_op.cc:301] APPLY: starting
0503 14:12:29.469587 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.469649 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.469684 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.469855 (+ 171us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.469899 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:29.469906 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:29.470020 (+ 114us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.477566 (+ 7546us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.477586 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:29.477592 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.477984 (+ 392us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.477989 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53779,"apply.run_cpu_time_us":716,"apply.run_wall_time_us":8509,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":755,"replication_time_us":636077,"thread_start_us":55537,"threads_started":1}]]}
I20240503 14:12:29.509107 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_960d64d809814618909c0ab5c4602156" } new_table_name: "default.Table_3c0e46c1611c4132ae4490057c568592" modify_external_catalogs: true
I20240503 14:12:29.478596 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 8cd037e135904e2cab09cbf39fed0b13
I20240503 14:12:29.511288 28600 tablet_bootstrap.cc:492] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.511438 28600 ts_tablet_manager.cc:1367] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.148s user 0.005s sys 0.009s
I20240503 14:12:29.511736 28032 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1225) took 1556 ms. Trace:
I20240503 14:12:29.511839 28032 rpcz_store.cc:276] 0503 14:12:27.955465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.740411 (+784946us) service_pool.cc:225] Handling call
0503 14:12:28.740821 (+ 410us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.740942 (+ 121us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:28.741023 (+ 81us) catalog_manager.cc:2758] Locking tablets
0503 14:12:28.783204 (+ 42181us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:29.478422 (+695218us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:29.478516 (+ 94us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:29.478564 (+ 48us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:29.511678 (+ 33114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.809083 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.809179 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.809184 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:28.809186 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.809286 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.809296 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.809301 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.809411 (+ 110us) tablet.cc:699] Row locks acquired
0503 14:12:28.809414 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.809453 (+ 39us) write_op.cc:270] Start()
0503 14:12:28.809486 (+ 33us) write_op.cc:276] Timestamp: P: 1714745548809426 usec, L: 0
0503 14:12:28.809490 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.809718 (+ 228us) log.cc:842] Serialized 762 byte log entry
0503 14:12:29.476341 (+666623us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.477101 (+ 760us) write_op.cc:301] APPLY: starting
0503 14:12:29.477116 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.477163 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.477197 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.477271 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.477304 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:29.477312 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:29.477373 (+ 61us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.478288 (+ 915us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.478306 (+ 18us) write_op.cc:454] Released schema lock
0503 14:12:29.478312 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.478372 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.478376 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":14080,"child_traces":[["op",{"apply.queue_time_us":699,"apply.run_cpu_time_us":552,"apply.run_wall_time_us":1329,"num_ops":2,"prepare.queue_time_us":25148,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":1068,"replication_time_us":666826,"spinlock_wait_cycles":1949952,"thread_start_us":5681,"threads_started":1}]]}
I20240503 14:12:29.512753 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5676e58199804bb9ad655f7fe44d0a9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:29.513375 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5676e58199804bb9ad655f7fe44d0a9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.515632 28600 raft_consensus.cc:348] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.515864 28600 raft_consensus.cc:374] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.515923 28600 raft_consensus.cc:729] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.516214 28600 consensus_queue.cc:260] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.485551 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 0f642b306a8c49fd9c070a9d340889a2
I20240503 14:12:29.516938 28030 catalog_manager.cc:6554] ReplaceTablet: tablet 0f642b306a8c49fd9c070a9d340889a2 deleted and replaced by tablet 4a28d2b3e5084a1b93f1074d100e397a
I20240503 14:12:29.517091 28030 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1222) took 1769 ms. Trace:
I20240503 14:12:29.517426 28600 ts_tablet_manager.cc:1398] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:29.517169 28030 rpcz_store.cc:276] 0503 14:12:27.748025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.738095 (+990070us) service_pool.cc:225] Handling call
0503 14:12:29.517035 (+778940us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.805837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.805970 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.805977 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:28.805980 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:28.806108 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.806121 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.806126 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.806205 (+ 79us) tablet.cc:699] Row locks acquired
0503 14:12:28.806208 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.806252 (+ 44us) write_op.cc:270] Start()
0503 14:12:28.806288 (+ 36us) write_op.cc:276] Timestamp: P: 1714745548806223 usec, L: 0
0503 14:12:28.806293 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.806527 (+ 234us) log.cc:842] Serialized 628 byte log entry
0503 14:12:29.471375 (+664848us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.471919 (+ 544us) write_op.cc:301] APPLY: starting
0503 14:12:29.471941 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.471996 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.472033 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.472199 (+ 166us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.472265 (+ 66us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:29.472273 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:29.472367 (+ 94us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.480935 (+ 8568us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.480959 (+ 24us) write_op.cc:454] Released schema lock
0503 14:12:29.480968 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.481039 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.481042 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":472,"apply.run_cpu_time_us":748,"apply.run_wall_time_us":9191,"num_ops":2,"prepare.queue_time_us":24193,"prepare.run_cpu_time_us":802,"prepare.run_wall_time_us":1671,"replication_time_us":665051,"thread_start_us":2501,"threads_started":1}]]}
I20240503 14:12:29.517971 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_b22b2e286534472a9988dfb13d282282"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:29.518538 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b22b2e286534472a9988dfb13d282282 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.519393 28614 tablet_bootstrap.cc:492] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.519543 28614 ts_tablet_manager.cc:1367] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.002s
I20240503 14:12:29.486172 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 310abf35ae02497c9d3696a0c49774ae
I20240503 14:12:29.520336 28614 raft_consensus.cc:348] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.520529 28614 raft_consensus.cc:374] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.520576 28614 raft_consensus.cc:729] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.520582 28031 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1227) took 1542 ms. Trace:
I20240503 14:12:29.520825 28614 consensus_queue.cc:260] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.520697 28031 rpcz_store.cc:276] 0503 14:12:27.978001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.770651 (+792650us) service_pool.cc:225] Handling call
0503 14:12:28.771083 (+ 432us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.771222 (+ 139us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:28.771339 (+ 117us) catalog_manager.cc:2758] Locking tablets
0503 14:12:28.784111 (+ 12772us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:29.485935 (+701824us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:29.486024 (+ 89us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:29.486081 (+ 57us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:29.520524 (+ 34443us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.810196 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.810298 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.810304 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.810306 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.810430 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.810440 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.810443 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.810537 (+ 94us) tablet.cc:699] Row locks acquired
0503 14:12:28.810540 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.810582 (+ 42us) write_op.cc:270] Start()
0503 14:12:28.810618 (+ 36us) write_op.cc:276] Timestamp: P: 1714745548810555 usec, L: 0
0503 14:12:28.810622 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.810802 (+ 180us) log.cc:842] Serialized 762 byte log entry
0503 14:12:29.482168 (+671366us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.482509 (+ 341us) write_op.cc:301] APPLY: starting
0503 14:12:29.482527 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.482576 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.482606 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.482682 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.482722 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:29.482730 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:29.482836 (+ 106us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.483130 (+ 294us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.483151 (+ 21us) write_op.cc:454] Released schema lock
0503 14:12:29.483158 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.483218 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.483223 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":293,"apply.run_cpu_time_us":555,"apply.run_wall_time_us":770,"num_ops":2,"prepare.queue_time_us":25467,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":1092,"replication_time_us":671520}]]}
I20240503 14:12:29.521490 28614 ts_tablet_manager.cc:1398] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:29.521493 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_076b0d07ce0c4cba9fbf355693ab1aa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:29.522064 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_076b0d07ce0c4cba9fbf355693ab1aa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.522163 28614 tablet_bootstrap.cc:492] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:29.522607 28290 consensus_peers.cc:485] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0175b38c1e0464a9a6208cfaf68ca0d. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:29.522863 28290 leader_election.cc:341] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 11e14e8b5f4046029af4673a277112d9
W20240503 14:12:29.523028 28290 leader_election.cc:341] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 11e14e8b5f4046029af4673a277112d9
I20240503 14:12:29.523104 28290 leader_election.cc:302] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:29.486549 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4b122081bd34391809a2c5746fad7e6 consensus state has no leader
I20240503 14:12:29.523468 28029 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1230) took 1525 ms. Trace:
I20240503 14:12:29.523555 28029 rpcz_store.cc:276] 0503 14:12:27.997753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.792737 (+794984us) service_pool.cc:225] Handling call
0503 14:12:28.793119 (+ 382us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.793312 (+ 193us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:28.793508 (+ 196us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:28.793603 (+ 95us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:29.486335 (+692732us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:29.523412 (+ 37077us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.811342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.811426 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.811431 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:12:28.811433 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.811518 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.811528 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.811533 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:28.811583 (+ 50us) tablet.cc:699] Row locks acquired
0503 14:12:28.811586 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.811618 (+ 32us) write_op.cc:270] Start()
0503 14:12:28.811646 (+ 28us) write_op.cc:276] Timestamp: P: 1714745548811592 usec, L: 0
0503 14:12:28.811650 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.811807 (+ 157us) log.cc:842] Serialized 405 byte log entry
0503 14:12:29.482240 (+670433us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.483315 (+ 1075us) write_op.cc:301] APPLY: starting
0503 14:12:29.483328 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.483419 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.483445 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.483487 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.483518 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:29.483526 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:29.483597 (+ 71us) log.cc:842] Serialized 23 byte log entry
0503 14:12:29.483825 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.483841 (+ 16us) write_op.cc:454] Released schema lock
0503 14:12:29.483847 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.483897 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.483901 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1045,"apply.run_cpu_time_us":428,"apply.run_wall_time_us":633,"num_ops":1,"prepare.queue_time_us":17175,"prepare.run_cpu_time_us":549,"prepare.run_wall_time_us":881,"replication_time_us":670578}]]}
I20240503 14:12:29.526665 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9a1bc119df7d4694bbe926611ef73245
I20240503 14:12:29.527083 28033 catalog_manager.cc:6554] ReplaceTablet: tablet 9a1bc119df7d4694bbe926611ef73245 deleted and replaced by tablet 7de83e2102924b8884fbf20300058f43
I20240503 14:12:29.527215 28033 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1231) took 1113 ms. Trace:
I20240503 14:12:29.527302 28033 rpcz_store.cc:276] 0503 14:12:28.413579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.797009 (+383430us) service_pool.cc:225] Handling call
0503 14:12:29.527162 (+730153us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.812259 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.812376 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.812382 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.812384 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.812488 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.812498 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.812503 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.812572 (+ 69us) tablet.cc:699] Row locks acquired
0503 14:12:28.812575 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.812610 (+ 35us) write_op.cc:270] Start()
0503 14:12:28.812638 (+ 28us) write_op.cc:276] Timestamp: P: 1714745548812584 usec, L: 0
0503 14:12:28.812642 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.812805 (+ 163us) log.cc:842] Serialized 628 byte log entry
0503 14:12:29.482280 (+669475us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.483983 (+ 1703us) write_op.cc:301] APPLY: starting
0503 14:12:29.483995 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.484029 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.484050 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.520890 (+ 36840us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.520959 (+ 69us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:29.520967 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:29.521066 (+ 99us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.521135 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.521158 (+ 23us) write_op.cc:454] Released schema lock
0503 14:12:29.521165 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.521231 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.521234 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1674,"apply.run_cpu_time_us":672,"apply.run_wall_time_us":37308,"num_ops":2,"prepare.queue_time_us":14072,"prepare.run_cpu_time_us":665,"prepare.run_wall_time_us":27953,"replication_time_us":669626}]]}
I20240503 14:12:29.528007 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2746fc24790d430ea83a7a159d11260e" } new_table_name: "default.Table_d236fdc90317450ba5b67738aad5e416" modify_external_catalogs: true
I20240503 14:12:29.528856 28614 tablet_bootstrap.cc:654] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.486837 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet dc31b2e907b04ea19a6f9f7fd8f57843
I20240503 14:12:29.529914 28027 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1226) took 1574 ms. Trace:
I20240503 14:12:29.530040 28027 rpcz_store.cc:276] 0503 14:12:27.955601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.766905 (+811304us) service_pool.cc:225] Handling call
0503 14:12:28.769398 (+ 2493us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedfca6 00007fe123ed1f66 00007fe123ed1c95 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:28.769753 (+ 355us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.769873 (+ 120us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:28.769954 (+ 81us) catalog_manager.cc:2758] Locking tablets
0503 14:12:28.826004 (+ 56050us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:29.486642 (+660638us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:29.486717 (+ 75us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:29.486763 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:29.529873 (+ 43110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.840275 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.840425 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.840431 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:28.840433 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.840556 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.840567 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.840571 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.840672 (+ 101us) tablet.cc:699] Row locks acquired
0503 14:12:28.840675 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.840720 (+ 45us) write_op.cc:270] Start()
0503 14:12:28.840757 (+ 37us) write_op.cc:276] Timestamp: P: 1714745548840688 usec, L: 0
0503 14:12:28.840761 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.857885 (+ 17124us) log.cc:842] Serialized 762 byte log entry
0503 14:12:29.482358 (+624473us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.484203 (+ 1845us) write_op.cc:301] APPLY: starting
0503 14:12:29.484215 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.484244 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.484267 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.484328 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.484357 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:29.484363 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:29.484423 (+ 60us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.484655 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.484671 (+ 16us) write_op.cc:454] Released schema lock
0503 14:12:29.484677 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.484764 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.484768 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5089536,"child_traces":[["op",{"apply.queue_time_us":1799,"apply.run_cpu_time_us":459,"apply.run_wall_time_us":616,"num_ops":2,"prepare.queue_time_us":13367,"prepare.run_cpu_time_us":941,"prepare.run_wall_time_us":17736,"replication_time_us":641583}]]}
I20240503 14:12:29.530830 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d501859ed20b479c84d1abf619d54d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:29.531361 28661 ts_tablet_manager.cc:1888] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:29.531423 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d501859ed20b479c84d1abf619d54d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.531489 28661 log.cc:1196] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/c816f3b05cfb4496984f6cb28d310cc4
I20240503 14:12:29.531925 30431 raft_consensus.cc:2726] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.487102 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet eff8442b6c02469eaedb08c51200829b
W20240503 14:12:29.532250 28290 leader_election.cc:341] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 9ac2833a0bb045dcb2824468d936ffe4
I20240503 14:12:29.532545 28026 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1217) took 1953 ms. Trace:
I20240503 14:12:29.533037 28661 ts_tablet_manager.cc:1909] T c816f3b05cfb4496984f6cb28d310cc4 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.532652 28026 rpcz_store.cc:276] 0503 14:12:27.579382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.633188 (+1053806us) service_pool.cc:225] Handling call
0503 14:12:28.661932 (+ 28744us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedfca6 00007fe123ed1f66 00007fe123ed1c95 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:28.671620 (+ 9688us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:28.671820 (+ 200us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:28.671891 (+ 71us) catalog_manager.cc:2758] Locking tablets
0503 14:12:28.827895 (+156004us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:29.486935 (+659040us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:29.487004 (+ 69us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:29.487040 (+ 36us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:29.532508 (+ 45468us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:28.858058 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:28.858189 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:28.858196 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:28.858198 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:28.858336 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:28.858351 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:28.858356 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:28.858493 (+ 137us) tablet.cc:699] Row locks acquired
0503 14:12:28.858496 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:28.858544 (+ 48us) write_op.cc:270] Start()
0503 14:12:28.858588 (+ 44us) write_op.cc:276] Timestamp: P: 1714745548858512 usec, L: 0
0503 14:12:28.858592 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:28.858808 (+ 216us) log.cc:842] Serialized 731 byte log entry
0503 14:12:29.482395 (+623587us) op_driver.cc:444] REPLICATION: finished
0503 14:12:29.484851 (+ 2456us) write_op.cc:301] APPLY: starting
0503 14:12:29.484862 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:29.484892 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:29.484913 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:29.484977 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:29.485005 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:29.485013 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:29.485096 (+ 83us) log.cc:842] Serialized 29 byte log entry
0503 14:12:29.485301 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.485316 (+ 15us) write_op.cc:454] Released schema lock
0503 14:12:29.485322 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:29.485436 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:29.485440 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":65439616,"child_traces":[["op",{"apply.queue_time_us":2428,"apply.run_cpu_time_us":433,"apply.run_wall_time_us":634,"num_ops":2,"prepare.queue_time_us":29469,"prepare.run_cpu_time_us":843,"prepare.run_wall_time_us":842,"replication_time_us":623791}]]}
I20240503 14:12:29.533548 28600 tablet_bootstrap.cc:492] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:29.533401 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0561c91abcaf4d26b3f20f19bbc4af18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:29.533977 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0561c91abcaf4d26b3f20f19bbc4af18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.535332 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 11e14e8b5f4046029af4673a277112d9 (table default.Table_68199a1d18244f54bb0b7b0256a58e2a [id=2528ba5e3fe74c40949dcc7c289d0630]) successfully deleted
I20240503 14:12:29.535584 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 9ac2833a0bb045dcb2824468d936ffe4 (table default.Table_472814febc0548599d68e796d57f48a1 [id=7128b9ce63544947ab63aa6d2490c691]) successfully deleted
I20240503 14:12:29.535743 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet c816f3b05cfb4496984f6cb28d310cc4 (table default.Table_94d6ea5674494c8a960eaee9c10ded44 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]) successfully deleted
I20240503 14:12:29.536724 28600 tablet_bootstrap.cc:654] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.539631 29828 ts_tablet_manager.cc:1888] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.543373 30404 ts_tablet_manager.cc:1909] T c816f3b05cfb4496984f6cb28d310cc4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:29.545444 29828 log.cc:1196] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/4e8411f474c247c2abfd0bd9682fb8a2
W20240503 14:12:29.542306 28290 leader_election.cc:341] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 9ac2833a0bb045dcb2824468d936ffe4
I20240503 14:12:29.546025 28290 leader_election.cc:302] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.546311 30431 raft_consensus.cc:2726] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.546806 30272 raft_consensus.cc:482] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.546965 30272 raft_consensus.cc:504] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.544639 28614 tablet_bootstrap.cc:492] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.547353 28614 ts_tablet_manager.cc:1367] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.004s
I20240503 14:12:29.547691 30272 leader_election.cc:288] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.548573 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b86451bc448428da7485e70e0d54b89" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.548672 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b86451bc448428da7485e70e0d54b89" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.549278 28150 leader_election.cc:341] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.549487 28150 leader_election.cc:341] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.549561 28150 leader_election.cc:302] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.549988 30272 raft_consensus.cc:2726] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.545890 29828 ts_tablet_manager.cc:1909] T 4e8411f474c247c2abfd0bd9682fb8a2 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.551878 30404 tablet_replica.cc:330] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:29.552110 30404 raft_consensus.cc:2229] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.552259 28614 raft_consensus.cc:348] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.552517 28614 raft_consensus.cc:374] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.552574 28614 raft_consensus.cc:729] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.552807 28614 consensus_queue.cc:260] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.559559 30404 raft_consensus.cc:2258] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.560875 30431 raft_consensus.cc:482] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.560982 28614 ts_tablet_manager.cc:1398] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:12:29.561035 30431 raft_consensus.cc:504] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.561301 28614 tablet_bootstrap.cc:492] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.561712 30431 leader_election.cc:288] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.562120 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.562393 28213 raft_consensus.cc:2379] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:29.562826 30404 ts_tablet_manager.cc:1875] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.562951 28614 tablet_bootstrap.cc:654] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.563432 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7261088abbc641db8d737a48bbac25bd" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.564551 28290 leader_election.cc:302] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.564797 30431 raft_consensus.cc:3037] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.567303 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1029) took 2654 ms. Trace:
I20240503 14:12:29.567415 30431 raft_consensus.cc:2726] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.567416 29828 rpcz_store.cc:276] 0503 14:12:26.913072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.916068 (+ 2996us) service_pool.cc:225] Handling call
0503 14:12:29.504207 (+2588139us) ts_tablet_manager.cc:1120] Deleting tablet 4e8411f474c247c2abfd0bd9682fb8a2
0503 14:12:29.504223 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.518222 (+ 13999us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.539487 (+ 21265us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.567275 (+ 27788us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2414,"tablet-delete.queue_time_us":2587800}
I20240503 14:12:29.569829 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1028) took 2656 ms. Trace:
I20240503 14:12:29.570003 29828 rpcz_store.cc:276] 0503 14:12:26.912972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.960946 (+ 47974us) service_pool.cc:225] Handling call
0503 14:12:29.569698 (+2608752us) ts_tablet_manager.cc:1120] Deleting tablet ff1958dc488e4854a8c8e74da9fab243
0503 14:12:29.569710 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.569808 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2608450}
I20240503 14:12:29.570591 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1030) took 2588 ms. Trace:
I20240503 14:12:29.570732 29828 rpcz_store.cc:276] 0503 14:12:26.982119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.982337 (+ 218us) service_pool.cc:225] Handling call
0503 14:12:29.570468 (+2588131us) ts_tablet_manager.cc:1120] Deleting tablet 9db657eef2db4e3791318e01e1a965fb
0503 14:12:29.570478 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.570575 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2587864}
I20240503 14:12:29.571260 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1031) took 2589 ms. Trace:
I20240503 14:12:29.571396 29828 rpcz_store.cc:276] 0503 14:12:26.982274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.986192 (+ 3918us) service_pool.cc:225] Handling call
0503 14:12:29.571141 (+2584949us) ts_tablet_manager.cc:1120] Deleting tablet 5ac145bcc44b4437b840df389cd254f9
0503 14:12:29.571151 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.571245 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2584644}
I20240503 14:12:29.571830 29828 tablet_replica.cc:330] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.572358 28604 tablet_bootstrap.cc:492] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:29.572522 28604 ts_tablet_manager.cc:1367] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.190s user 0.011s sys 0.015s
I20240503 14:12:29.572700 29828 raft_consensus.cc:2229] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:29.573139 29828 raft_consensus.cc:2258] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.573256 28604 raft_consensus.cc:348] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.573472 28604 raft_consensus.cc:374] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.573534 28604 raft_consensus.cc:729] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.574064 29828 ts_tablet_manager.cc:1875] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.575968 28604 consensus_queue.cc:260] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.577708 28604 ts_tablet_manager.cc:1398] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:29.578713 28604 tablet_bootstrap.cc:492] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:29.579247 30272 raft_consensus.cc:482] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.579398 30272 raft_consensus.cc:504] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.579819 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.580116 30272 leader_election.cc:288] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.580193 28661 tablet_replica.cc:330] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.580400 28661 raft_consensus.cc:2229] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.580549 28661 raft_consensus.cc:2258] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.580840 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.581094 28661 ts_tablet_manager.cc:1875] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.581467 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.581631 28614 tablet_bootstrap.cc:492] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.581832 28614 ts_tablet_manager.cc:1367] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.002s
W20240503 14:12:29.582608 28150 leader_election.cc:341] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.582861 28150 leader_election.cc:341] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.582935 28150 leader_election.cc:302] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.583806 30272 raft_consensus.cc:2726] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.585114 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.585616 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df with delete_type TABLET_DATA_DELETED (replaced by tablet ae909ab6263a401490952e3f68533741) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.585783 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.585934 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.586162 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.586881 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.587242 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:29.587509 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet c4b122081bd34391809a2c5746fad7e6 (table default.Table_8668db655be7429dbdc3ee1cd7512979 [id=4387686ed640413aa8ac41dcad75af40], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c4b122081bd34391809a2c5746fad7e6 consensus state has no leader
I20240503 14:12:29.587895 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.582576 28614 raft_consensus.cc:348] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.588199 28614 raft_consensus.cc:374] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.588287 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.588299 28614 raft_consensus.cc:729] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.588752 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f1f5216cc940888803ce8f6acd2a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.588616 28614 consensus_queue.cc:260] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.589149 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f1f5216cc940888803ce8f6acd2a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.589215 28614 ts_tablet_manager.cc:1398] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:12:29.589442 28600 tablet_bootstrap.cc:492] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.589496 28614 tablet_bootstrap.cc:492] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.589594 28604 tablet_bootstrap.cc:654] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.589589 28600 ts_tablet_manager.cc:1367] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.003s
I20240503 14:12:29.591241 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f1f5216cc940888803ce8f6acd2a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:29.591748 28600 raft_consensus.cc:348] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.591969 28600 raft_consensus.cc:374] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.592028 28600 raft_consensus.cc:729] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.592466 30446 raft_consensus.cc:482] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.592612 30446 raft_consensus.cc:504] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.593255 30446 leader_election.cc:288] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.593587 28614 tablet_bootstrap.cc:654] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.592423 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd037e135904e2cab09cbf39fed0b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:29.593997 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee9b69dd1f47219e7d3126207810c7" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.593999 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee9b69dd1f47219e7d3126207810c7" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.594074 28600 consensus_queue.cc:260] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.594259 28213 raft_consensus.cc:2454] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.594791 28600 ts_tablet_manager.cc:1398] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20240503 14:12:29.595242 28427 leader_election.cc:341] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.596010 28427 leader_election.cc:302] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:29.596143 28600 tablet_bootstrap.cc:492] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:29.596544 30446 raft_consensus.cc:2781] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.596628 30446 raft_consensus.cc:482] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.596688 30446 raft_consensus.cc:3037] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.596741 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd037e135904e2cab09cbf39fed0b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.599524 30446 raft_consensus.cc:504] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.600896 28600 tablet_bootstrap.cc:654] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.601459 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1bc119df7d4694bbe926611ef73245 with delete_type TABLET_DATA_DELETED (replaced by tablet 7de83e2102924b8884fbf20300058f43) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:29.601457 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f642b306a8c49fd9c070a9d340889a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a28d2b3e5084a1b93f1074d100e397a) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:29.601508 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet 310abf35ae02497c9d3696a0c49774ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:29.601933 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet eff8442b6c02469eaedb08c51200829b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:29.602088 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet dc31b2e907b04ea19a6f9f7fd8f57843 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:29.602339 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 20b176a071ba4f3fbff2a64d6e46027a (table default.Table_89c30ada87fb4dd18b7e0acaf2c7fa82 [id=2cb8414c296e4f9898bf3977e6e5d80e]) successfully deleted
I20240503 14:12:29.602684 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 310abf35ae02497c9d3696a0c49774ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.602805 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet dc31b2e907b04ea19a6f9f7fd8f57843 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.602957 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet eff8442b6c02469eaedb08c51200829b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.603024 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1bc119df7d4694bbe926611ef73245 with delete_type TABLET_DATA_DELETED (replaced by tablet 7de83e2102924b8884fbf20300058f43) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.603051 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee9b69dd1f47219e7d3126207810c7" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:29.603839 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f642b306a8c49fd9c070a9d340889a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a28d2b3e5084a1b93f1074d100e397a) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:29.604727 28427 leader_election.cc:341] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.605005 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd037e135904e2cab09cbf39fed0b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.605638 30446 leader_election.cc:288] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.605867 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f642b306a8c49fd9c070a9d340889a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a28d2b3e5084a1b93f1074d100e397a) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.606345 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 310abf35ae02497c9d3696a0c49774ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.606771 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1bc119df7d4694bbe926611ef73245 with delete_type TABLET_DATA_DELETED (replaced by tablet 7de83e2102924b8884fbf20300058f43) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.607079 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet dc31b2e907b04ea19a6f9f7fd8f57843 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.607460 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet eff8442b6c02469eaedb08c51200829b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:28 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:29.608182 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee9b69dd1f47219e7d3126207810c7" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.608400 28213 raft_consensus.cc:3037] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.611212 28213 raft_consensus.cc:2454] T 33ee9b69dd1f47219e7d3126207810c7 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.612813 28427 leader_election.cc:302] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:29.613552 30335 raft_consensus.cc:2781] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.613701 30335 raft_consensus.cc:686] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.613960 30335 consensus_queue.cc:237] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.615334 28661 ts_tablet_manager.cc:1888] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.615449 28661 log.cc:1196] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/007a892ef8504b848a400c05740eaa0c
I20240503 14:12:29.615734 28661 ts_tablet_manager.cc:1909] T 007a892ef8504b848a400c05740eaa0c P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.616739 28661 tablet_replica.cc:330] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.616961 28661 raft_consensus.cc:2229] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:29.617214 28661 pending_rounds.cc:62] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:29.617273 28661 pending_rounds.cc:68] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597767654764544 op_type: NO_OP noop_request { }
I20240503 14:12:29.617396 28661 raft_consensus.cc:2866] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:29.617475 28661 raft_consensus.cc:2258] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.621718 28600 tablet_bootstrap.cc:492] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.621893 28600 ts_tablet_manager.cc:1367] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.004s
I20240503 14:12:29.622537 29828 ts_tablet_manager.cc:1888] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.622650 29828 log.cc:1196] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/e0175b38c1e0464a9a6208cfaf68ca0d
I20240503 14:12:29.622778 28600 raft_consensus.cc:348] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.622982 29828 ts_tablet_manager.cc:1909] T e0175b38c1e0464a9a6208cfaf68ca0d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.623000 28600 raft_consensus.cc:374] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.623054 28600 raft_consensus.cc:729] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.623679 28600 consensus_queue.cc:260] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.625752 28661 ts_tablet_manager.cc:1875] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.626165 28600 ts_tablet_manager.cc:1398] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:29.626487 28600 tablet_bootstrap.cc:492] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:29.632411 28600 tablet_bootstrap.cc:654] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.635310 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1032) took 2355 ms. Trace:
I20240503 14:12:29.635887 29828 rpcz_store.cc:276] 0503 14:12:27.280071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.326245 (+ 46174us) service_pool.cc:225] Handling call
0503 14:12:29.571778 (+2245533us) ts_tablet_manager.cc:1120] Deleting tablet e0175b38c1e0464a9a6208cfaf68ca0d
0503 14:12:29.571788 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.619410 (+ 47622us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.622424 (+ 3014us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.635281 (+ 12857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245262}
W20240503 14:12:29.636595 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2221218847554cbd9d3242a9a6431e46 because the tablet was not found. No further retry: Not found: Tablet not found: 2221218847554cbd9d3242a9a6431e46
W20240503 14:12:29.637027 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7b7ca19dd15f48a68ace7fefc7ae660f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b7ca19dd15f48a68ace7fefc7ae660f already in progress: creating tablet
W20240503 14:12:29.637293 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7d348da459634561985c4ac1710c0bb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d348da459634561985c4ac1710c0bb2 already in progress: creating tablet
I20240503 14:12:29.637475 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 5b90e9563b4c4cef8ee7964eaa96e285 (table default.Table_4d12dbb4326d40ddbeb81be96c45901b [id=11af424ea24549bc83f6203448ae619c]) successfully deleted
W20240503 14:12:29.637712 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 728ba3ac1cb645168481b84b5b548140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728ba3ac1cb645168481b84b5b548140 already in progress: creating tablet
W20240503 14:12:29.637918 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet bab214b7ec58425d8cd1180ee16b273d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab214b7ec58425d8cd1180ee16b273d already in progress: creating tablet
W20240503 14:12:29.638131 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c725628f72884086b523b56ecf876069 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c725628f72884086b523b56ecf876069 already in progress: creating tablet
I20240503 14:12:29.638309 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 3d54cc5c03034a6685c1da34f9bd65ac (table default.Table_8f505fa519514278ae7489b9ff7bbaf2 [id=f58e9b32e96948cfab2760b64eb18b68]) successfully deleted
I20240503 14:12:29.638794 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 4e8411f474c247c2abfd0bd9682fb8a2 (table default.Table_fc189ad8c19041f7814fac6de9f482cb [id=b7f4f49135e245dc975ffd7eb0589a8d]) successfully deleted
I20240503 14:12:29.639735 30272 raft_consensus.cc:482] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.639909 30272 raft_consensus.cc:504] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.640619 30272 leader_election.cc:288] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.641621 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac51b83abe14ab5b1fb5d627233e58e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.641786 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac51b83abe14ab5b1fb5d627233e58e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.642508 28150 leader_election.cc:341] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:29.642719 28150 leader_election.cc:341] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.642791 28150 leader_election.cc:302] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.643298 30272 raft_consensus.cc:2726] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.643819 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1034) took 2317 ms. Trace:
I20240503 14:12:29.643985 29828 rpcz_store.cc:276] 0503 14:12:27.326509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.326555 (+ 46us) service_pool.cc:225] Handling call
0503 14:12:29.643656 (+2317101us) ts_tablet_manager.cc:1120] Deleting tablet a526e9f1e9934e519349dcd4523ee694
0503 14:12:29.643671 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.643797 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2316931}
I20240503 14:12:29.644745 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1033) took 2318 ms. Trace:
I20240503 14:12:29.644903 29828 rpcz_store.cc:276] 0503 14:12:27.326388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.326556 (+ 168us) service_pool.cc:225] Handling call
0503 14:12:29.644614 (+2318058us) ts_tablet_manager.cc:1120] Deleting tablet e04c0b6955b34267820ceecfffcfb88a
0503 14:12:29.644625 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.644727 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23,"tablet-delete.queue_time_us":2317888}
I20240503 14:12:29.645429 29828 tablet_replica.cc:330] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.645709 29828 raft_consensus.cc:2229] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.646371 29828 raft_consensus.cc:2258] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.646874 28600 tablet_bootstrap.cc:492] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.647028 28600 ts_tablet_manager.cc:1367] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 14:12:29.647799 28600 raft_consensus.cc:348] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.648000 28600 raft_consensus.cc:374] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.648217 29828 ts_tablet_manager.cc:1875] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.648046 28600 raft_consensus.cc:729] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.644784 28614 tablet_bootstrap.cc:492] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.648847 28600 consensus_queue.cc:260] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.649258 28614 ts_tablet_manager.cc:1367] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.007s
I20240503 14:12:29.650826 28600 ts_tablet_manager.cc:1398] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:29.651127 28600 tablet_bootstrap.cc:492] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:29.659279 28600 tablet_bootstrap.cc:654] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.661199 28614 raft_consensus.cc:348] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.661355 28604 tablet_bootstrap.cc:492] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:29.661553 28604 ts_tablet_manager.cc:1367] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20240503 14:12:29.662436 28604 raft_consensus.cc:348] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.662734 28604 raft_consensus.cc:374] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.662798 28604 raft_consensus.cc:729] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.662935 28614 raft_consensus.cc:374] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.663077 28661 ts_tablet_manager.cc:1888] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.663200 28614 raft_consensus.cc:729] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
W20240503 14:12:29.639077 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ff1958dc488e4854a8c8e74da9fab243 because the tablet was not found. No further retry: Not found: Tablet not found: ff1958dc488e4854a8c8e74da9fab243
I20240503 14:12:29.663040 28604 consensus_queue.cc:260] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.663677 28614 consensus_queue.cc:260] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.664480 28604 ts_tablet_manager.cc:1398] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:29.664844 28614 ts_tablet_manager.cc:1398] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:12:29.665019 28604 tablet_bootstrap.cc:492] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:29.666137 28614 tablet_bootstrap.cc:492] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.668179 30272 raft_consensus.cc:482] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.668360 30272 raft_consensus.cc:504] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.669067 30272 leader_election.cc:288] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.669580 28614 tablet_bootstrap.cc:654] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.670351 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996bb3a7edfc4f5a8a3a39ea6fcfe5cd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:29.671029 28150 leader_election.cc:341] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.671521 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996bb3a7edfc4f5a8a3a39ea6fcfe5cd" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.672680 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9db657eef2db4e3791318e01e1a965fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9db657eef2db4e3791318e01e1a965fb already in progress: creating tablet
W20240503 14:12:29.673038 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5ac145bcc44b4437b840df389cd254f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ac145bcc44b4437b840df389cd254f9 already in progress: creating tablet
I20240503 14:12:29.673072 29828 ts_tablet_manager.cc:1888] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.673192 29828 log.cc:1196] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0a50b3e199314bfc8465fdb7b2425d07
I20240503 14:12:29.673527 29828 ts_tablet_manager.cc:1909] T 0a50b3e199314bfc8465fdb7b2425d07 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:29.673604 28150 leader_election.cc:341] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.673731 28150 leader_election.cc:302] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.674304 30272 raft_consensus.cc:2726] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.674552 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1036) took 2347 ms. Trace:
I20240503 14:12:29.674659 29828 rpcz_store.cc:276] 0503 14:12:27.326926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.326962 (+ 36us) service_pool.cc:225] Handling call
0503 14:12:29.645377 (+2318415us) ts_tablet_manager.cc:1120] Deleting tablet 0a50b3e199314bfc8465fdb7b2425d07
0503 14:12:29.645388 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.655111 (+ 9723us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.672941 (+ 17830us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.674520 (+ 1579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2318264}
I20240503 14:12:29.663388 28661 log.cc:1196] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/2cad43f3c6094253ba87d0ed3fc29fd9
I20240503 14:12:29.676962 28661 ts_tablet_manager.cc:1909] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.677616 30404 ts_tablet_manager.cc:1888] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.677821 30404 log.cc:1196] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/49445f0f1dd1451c9ba066e61273f2f1
I20240503 14:12:29.678215 30404 ts_tablet_manager.cc:1909] T 49445f0f1dd1451c9ba066e61273f2f1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:29.678369 28427 consensus_peers.cc:485] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.679832 30404 tablet_replica.cc:330] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:29.680181 30404 raft_consensus.cc:2229] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.680402 30404 raft_consensus.cc:2258] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.681229 29828 tablet_replica.cc:330] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.681003 30404 ts_tablet_manager.cc:1875] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.681464 29828 raft_consensus.cc:2229] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.681655 29828 raft_consensus.cc:2258] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.682286 29828 ts_tablet_manager.cc:1875] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.683923 28614 tablet_bootstrap.cc:492] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.684101 28614 ts_tablet_manager.cc:1367] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 14:12:29.684907 28614 raft_consensus.cc:348] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.685109 28614 raft_consensus.cc:374] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.685206 28614 raft_consensus.cc:729] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.685624 28614 consensus_queue.cc:260] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.686197 28614 ts_tablet_manager.cc:1398] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:29.686853 28614 tablet_bootstrap.cc:492] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.687800 28661 tablet_replica.cc:330] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:29.688198 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet e0175b38c1e0464a9a6208cfaf68ca0d (table default.Table_7fdf02629122402c9cbf8053f89eced3 [id=5201e94e1d164116b74d4329e5991eca]) successfully deleted
I20240503 14:12:29.688875 28614 tablet_bootstrap.cc:654] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:29.689553 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a526e9f1e9934e519349dcd4523ee694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a526e9f1e9934e519349dcd4523ee694 already in progress: creating tablet
W20240503 14:12:29.689918 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e04c0b6955b34267820ceecfffcfb88a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e04c0b6955b34267820ceecfffcfb88a already in progress: creating tablet
I20240503 14:12:29.690176 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0a50b3e199314bfc8465fdb7b2425d07 (table default.Table_685e7e52f1744b14bdcb0c2d9f87c2a0 [id=82ad92c83a0c44bc86486fb6137a47e3]) successfully deleted
I20240503 14:12:29.690992 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet c816f3b05cfb4496984f6cb28d310cc4 (table default.Table_94d6ea5674494c8a960eaee9c10ded44 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]) successfully deleted
I20240503 14:12:29.691298 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 49445f0f1dd1451c9ba066e61273f2f1 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
W20240503 14:12:29.691704 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 920b41e3b21347b393497de4d4e87469 because the tablet was not found. No further retry: Not found: Tablet not found: 920b41e3b21347b393497de4d4e87469
I20240503 14:12:29.692766 30431 consensus_queue.cc:1035] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:29.702253 28604 tablet_bootstrap.cc:654] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.702803 28661 raft_consensus.cc:2229] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.704427 28661 raft_consensus.cc:2258] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.705147 28661 ts_tablet_manager.cc:1875] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.708835 29828 ts_tablet_manager.cc:1888] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.708982 29828 log.cc:1196] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/c9325d5142af4b018354bf38370b61e0
I20240503 14:12:29.709342 29828 ts_tablet_manager.cc:1909] T c9325d5142af4b018354bf38370b61e0 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.710196 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1037) took 2154 ms. Trace:
I20240503 14:12:29.710319 29828 rpcz_store.cc:276] 0503 14:12:27.555515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.555569 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:29.675136 (+2119567us) ts_tablet_manager.cc:1120] Deleting tablet c9325d5142af4b018354bf38370b61e0
0503 14:12:29.675468 (+ 332us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.704176 (+ 28708us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.708708 (+ 4532us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.710167 (+ 1459us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":2119337}
I20240503 14:12:29.711345 28614 tablet_bootstrap.cc:492] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.711489 28614 ts_tablet_manager.cc:1367] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.001s
I20240503 14:12:29.712327 28614 raft_consensus.cc:348] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.712527 28614 raft_consensus.cc:374] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.712569 28614 raft_consensus.cc:729] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.712769 28614 consensus_queue.cc:260] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.708948 28600 tablet_bootstrap.cc:492] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.714954 28600 ts_tablet_manager.cc:1367] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.008s
I20240503 14:12:29.715951 28600 raft_consensus.cc:348] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.716198 28600 raft_consensus.cc:374] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.716269 28600 raft_consensus.cc:729] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.716492 28600 consensus_queue.cc:260] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.717119 28600 ts_tablet_manager.cc:1398] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:29.717440 28600 tablet_bootstrap.cc:492] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:29.717451 28604 tablet_bootstrap.cc:492] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:29.717651 28604 ts_tablet_manager.cc:1367] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.001s
I20240503 14:12:29.718657 28604 raft_consensus.cc:348] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.720304 28604 raft_consensus.cc:374] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.720428 28604 raft_consensus.cc:729] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.720724 28604 consensus_queue.cc:260] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.722458 28604 ts_tablet_manager.cc:1398] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:12:29.723302 28604 tablet_bootstrap.cc:492] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:29.725018 28614 ts_tablet_manager.cc:1398] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:29.725487 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1038) took 2169 ms. Trace:
I20240503 14:12:29.725678 29828 rpcz_store.cc:276] 0503 14:12:27.555631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.560385 (+ 4754us) service_pool.cc:225] Handling call
0503 14:12:29.725258 (+2164873us) ts_tablet_manager.cc:1120] Deleting tablet b97da778da214bb6bc147c5ada6c7603
0503 14:12:29.725267 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.725463 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2152451}
I20240503 14:12:29.730278 28600 tablet_bootstrap.cc:654] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.693514 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 007a892ef8504b848a400c05740eaa0c (table default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]) successfully deleted
I20240503 14:12:29.731974 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 2cad43f3c6094253ba87d0ed3fc29fd9 (table default.Table_a9e8bd96c6824e9c8c64036f0848c876 [id=9e438ac1c3254c7aae6f81c707f433d7]) successfully deleted
W20240503 14:12:29.737589 28290 consensus_peers.cc:485] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62e4f6e3908e420e9fa156ee24b437d6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.739658 28614 tablet_bootstrap.cc:492] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.741923 28614 tablet_bootstrap.cc:654] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.743464 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1039) took 2182 ms. Trace:
I20240503 14:12:29.743589 29828 rpcz_store.cc:276] 0503 14:12:27.560897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.560939 (+ 42us) service_pool.cc:225] Handling call
0503 14:12:29.743282 (+2182343us) ts_tablet_manager.cc:1120] Deleting tablet f91ddbc250c44fbfbf4ffe83433d8dd8
0503 14:12:29.743293 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.743435 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2182195}
I20240503 14:12:29.744001 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1051) took 1780 ms. Trace:
I20240503 14:12:29.744102 29828 rpcz_store.cc:276] 0503 14:12:27.963982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.964030 (+ 48us) service_pool.cc:225] Handling call
0503 14:12:29.743887 (+1779857us) ts_tablet_manager.cc:1120] Deleting tablet 7549b2ca31b04247bf3a2202e7585817
0503 14:12:29.743898 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.743986 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1779570}
I20240503 14:12:29.744436 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1052) took 1780 ms. Trace:
I20240503 14:12:29.744513 29828 rpcz_store.cc:276] 0503 14:12:27.964103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.968158 (+ 4055us) service_pool.cc:225] Handling call
0503 14:12:29.744335 (+1776177us) ts_tablet_manager.cc:1120] Deleting tablet 3dfd6d40a57e4bea973874ff12c0ea3a
0503 14:12:29.744343 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.744421 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775933}
I20240503 14:12:29.744810 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1053) took 1780 ms. Trace:
I20240503 14:12:29.744886 29828 rpcz_store.cc:276] 0503 14:12:27.964214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:27.968449 (+ 4235us) service_pool.cc:225] Handling call
0503 14:12:29.744717 (+1776268us) ts_tablet_manager.cc:1120] Deleting tablet 5500da0e5658492b840dd29273ab265f
0503 14:12:29.744726 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.744796 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776134}
I20240503 14:12:29.745174 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1054) took 1732 ms. Trace:
I20240503 14:12:29.745255 29828 rpcz_store.cc:276] 0503 14:12:28.012497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.012657 (+ 160us) service_pool.cc:225] Handling call
0503 14:12:29.745087 (+1732430us) ts_tablet_manager.cc:1120] Deleting tablet 5685a6b3ea6943f8a72d74de87ac2166
0503 14:12:29.745094 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.745161 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727925}
I20240503 14:12:29.745543 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1055) took 1732 ms. Trace:
I20240503 14:12:29.745613 29828 rpcz_store.cc:276] 0503 14:12:28.012678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.012879 (+ 201us) service_pool.cc:225] Handling call
0503 14:12:29.745459 (+1732580us) ts_tablet_manager.cc:1120] Deleting tablet 91eb4433685b4119a170054a0a777e05
0503 14:12:29.745466 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.745530 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1728183}
I20240503 14:12:29.745862 29828 tablet_replica.cc:330] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.746091 29828 raft_consensus.cc:2229] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.746294 29828 raft_consensus.cc:2258] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.746964 29828 ts_tablet_manager.cc:1875] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.750154 30272 raft_consensus.cc:482] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.750321 30272 raft_consensus.cc:504] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.750964 30272 leader_election.cc:288] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.751284 28614 tablet_bootstrap.cc:492] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.751428 28614 ts_tablet_manager.cc:1367] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:12:29.751892 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e220336186f648419b186feada6f0124" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.752440 28614 raft_consensus.cc:348] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.752614 28614 raft_consensus.cc:374] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.752658 28614 raft_consensus.cc:729] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.752907 28614 consensus_queue.cc:260] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:29.753019 28150 leader_election.cc:341] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.753724 28614 ts_tablet_manager.cc:1398] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:29.753891 29828 ts_tablet_manager.cc:1888] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.753988 29828 log.cc:1196] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a884cf4ea11e48ef941722b79ad50075
I20240503 14:12:29.754026 28614 tablet_bootstrap.cc:492] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.754348 29828 ts_tablet_manager.cc:1909] T a884cf4ea11e48ef941722b79ad50075 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:29.755105 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1056) took 1742 ms. Trace:
I20240503 14:12:29.755188 29828 rpcz_store.cc:276] 0503 14:12:28.012802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.012855 (+ 53us) service_pool.cc:225] Handling call
0503 14:12:29.745811 (+1732956us) ts_tablet_manager.cc:1120] Deleting tablet a884cf4ea11e48ef941722b79ad50075
0503 14:12:29.745819 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.750654 (+ 4835us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.753784 (+ 3130us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.755084 (+ 1300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1728475}
I20240503 14:12:29.755604 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1061) took 1069 ms. Trace:
I20240503 14:12:29.755690 29828 rpcz_store.cc:276] 0503 14:12:28.685699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.685781 (+ 82us) service_pool.cc:225] Handling call
0503 14:12:29.755475 (+1069694us) ts_tablet_manager.cc:1120] Deleting tablet e220336186f648419b186feada6f0124
0503 14:12:29.755485 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.755588 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068850}
I20240503 14:12:29.755959 29828 tablet_replica.cc:330] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.756165 29828 raft_consensus.cc:2229] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.756177 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet c9325d5142af4b018354bf38370b61e0 (table default.Table_2ce3b6f2b77b4607856fc2a64ead5bb5 [id=db393a7e2a83466385ffc0af3d2789c4]) successfully deleted
I20240503 14:12:29.756314 29828 pending_rounds.cc:62] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:29.756354 29828 pending_rounds.cc:68] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597763886944256 op_type: NO_OP noop_request { }
I20240503 14:12:29.756453 28614 tablet_bootstrap.cc:654] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.756462 29828 raft_consensus.cc:2866] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:29.756587 29828 raft_consensus.cc:2258] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:29.756567 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet b97da778da214bb6bc147c5ada6c7603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b97da778da214bb6bc147c5ada6c7603 already in progress: creating tablet
W20240503 14:12:29.756795 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f91ddbc250c44fbfbf4ffe83433d8dd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f91ddbc250c44fbfbf4ffe83433d8dd8 already in progress: creating tablet
W20240503 14:12:29.757051 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7549b2ca31b04247bf3a2202e7585817 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7549b2ca31b04247bf3a2202e7585817 already in progress: creating tablet
I20240503 14:12:29.757316 29828 ts_tablet_manager.cc:1875] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:29.760442 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 3dfd6d40a57e4bea973874ff12c0ea3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dfd6d40a57e4bea973874ff12c0ea3a already in progress: creating tablet
I20240503 14:12:29.763645 29828 ts_tablet_manager.cc:1888] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:29.763785 29828 log.cc:1196] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/dbcc6cef7b7d464a9fbbae61caf0437d
I20240503 14:12:29.764153 29828 ts_tablet_manager.cc:1909] T dbcc6cef7b7d464a9fbbae61caf0437d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:29.768000 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5500da0e5658492b840dd29273ab265f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5500da0e5658492b840dd29273ab265f already in progress: creating tablet
W20240503 14:12:29.768522 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5685a6b3ea6943f8a72d74de87ac2166 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5685a6b3ea6943f8a72d74de87ac2166 already in progress: creating tablet
W20240503 14:12:29.768913 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 91eb4433685b4119a170054a0a777e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91eb4433685b4119a170054a0a777e05 already in progress: creating tablet
I20240503 14:12:29.769228 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a884cf4ea11e48ef941722b79ad50075 (table default.Table_cc77503fb1ad4d03a6278aef9f3ab979 [id=4b3b8bed2fba4370a246902bec7554f7]) successfully deleted
W20240503 14:12:29.769588 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e220336186f648419b186feada6f0124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e220336186f648419b186feada6f0124 already in progress: creating tablet
I20240503 14:12:29.758147 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e220336186f648419b186feada6f0124" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.781961 28150 leader_election.cc:341] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.782127 28150 leader_election.cc:302] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.782555 30272 raft_consensus.cc:2726] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.796439 28614 tablet_bootstrap.cc:492] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.796658 28614 ts_tablet_manager.cc:1367] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.004s
I20240503 14:12:29.797595 28614 raft_consensus.cc:348] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.797860 28614 raft_consensus.cc:374] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.797937 28614 raft_consensus.cc:729] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.798316 28614 consensus_queue.cc:260] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.799331 28614 ts_tablet_manager.cc:1398] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:29.803660 28614 tablet_bootstrap.cc:492] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.805577 28614 tablet_bootstrap.cc:654] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.809178 28600 tablet_bootstrap.cc:492] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.809355 28600 ts_tablet_manager.cc:1367] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.092s user 0.011s sys 0.006s
I20240503 14:12:29.810216 28600 raft_consensus.cc:348] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.810424 28600 raft_consensus.cc:374] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.810469 28600 raft_consensus.cc:729] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.810683 28600 consensus_queue.cc:260] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.811241 28600 ts_tablet_manager.cc:1398] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:29.811565 28600 tablet_bootstrap.cc:492] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:29.813786 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5bf863dabf6647618732a03af84efab3 consensus state has no leader
W20240503 14:12:29.814384 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74d5aaf39f1c4a5bb3f9f0c9773b0489 consensus state has no leader
I20240503 14:12:29.814523 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cb05adc8775b48d1837187d135ce1201"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:29.815153 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb05adc8775b48d1837187d135ce1201 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.816802 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_006d60d6900a406b8a5c73a795718f63" } new_table_name: "default.Table_54942d8495b94e56b3ab48580faf5e3b" modify_external_catalogs: true
I20240503 14:12:29.818262 30272 raft_consensus.cc:482] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.818432 30272 raft_consensus.cc:504] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.819166 30272 leader_election.cc:288] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.819249 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c25685742fa44084940ddfc0a0b5cb9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:29.819415 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_bfa7ab95de17487db85d9c7e73e24e4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:29.819942 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c25685742fa44084940ddfc0a0b5cb9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.820333 28600 tablet_bootstrap.cc:654] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.821113 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_392e46b937cf45e58d01188a21cf276c" } new_table_name: "default.Table_ec97037ea5a849bfb48a72d6771fbf1f" modify_external_catalogs: true
W20240503 14:12:29.823135 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfa7ab95de17487db85d9c7e73e24e4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.823839 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:29.824577 28150 leader_election.cc:341] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.828824 28026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_79162d4c00234c2eacb31954696e0e74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:29.829532 28026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79162d4c00234c2eacb31954696e0e74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:29.832330 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
I20240503 14:12:29.832835 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:29.833113 28024 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:29.838562 28029 catalog_manager.cc:5484] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.839315 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:29.839640 28029 catalog_manager.cc:5484] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.843714 30404 ts_tablet_manager.cc:1888] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.843850 30404 log.cc:1196] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/007a892ef8504b848a400c05740eaa0c
I20240503 14:12:29.844272 30404 ts_tablet_manager.cc:1909] T 007a892ef8504b848a400c05740eaa0c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:29.848533 28600 tablet_bootstrap.cc:492] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.848737 28600 ts_tablet_manager.cc:1367] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.002s
I20240503 14:12:29.850522 28600 raft_consensus.cc:348] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.850787 28600 raft_consensus.cc:374] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.850843 28600 raft_consensus.cc:729] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.851387 28600 consensus_queue.cc:260] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.840070 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
I20240503 14:12:29.851871 28614 tablet_bootstrap.cc:492] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.852021 28614 ts_tablet_manager.cc:1367] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20240503 14:12:29.851956 28029 catalog_manager.cc:5484] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.852466 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (table default.Table_bfff85704d88463383cbf94f9b121359 [id=bd1950deb4274d86bf288d501cf5c710]). Expected version 2 got 0
I20240503 14:12:29.853211 28614 raft_consensus.cc:348] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.853417 28614 raft_consensus.cc:374] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.853482 28614 raft_consensus.cc:729] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.853737 28614 consensus_queue.cc:260] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.854418 28614 ts_tablet_manager.cc:1398] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:29.854717 28614 tablet_bootstrap.cc:492] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.845271 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet dbcc6cef7b7d464a9fbbae61caf0437d (table default.Table_7c4e797497e74bbe98ff720b9ffd05ff [id=ee4e5312119d414aa8f8d01c28e0b5fb]) successfully deleted
W20240503 14:12:29.855363 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 09f1f5216cc940888803ce8f6acd2a47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09f1f5216cc940888803ce8f6acd2a47 already in progress: creating tablet
W20240503 14:12:29.855643 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 8cd037e135904e2cab09cbf39fed0b13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cd037e135904e2cab09cbf39fed0b13 already in progress: creating tablet
W20240503 14:12:29.855888 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 310abf35ae02497c9d3696a0c49774ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 310abf35ae02497c9d3696a0c49774ae already in progress: creating tablet
W20240503 14:12:29.856137 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet dc31b2e907b04ea19a6f9f7fd8f57843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc31b2e907b04ea19a6f9f7fd8f57843 already in progress: creating tablet
W20240503 14:12:29.856427 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet eff8442b6c02469eaedb08c51200829b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff8442b6c02469eaedb08c51200829b already in progress: creating tablet
W20240503 14:12:29.856712 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9a1bc119df7d4694bbe926611ef73245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a1bc119df7d4694bbe926611ef73245 already in progress: creating tablet
I20240503 14:12:29.858042 28614 tablet_bootstrap.cc:654] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.859160 28029 catalog_manager.cc:5484] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.862663 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2]). Expected version 3 got 0
I20240503 14:12:29.868512 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 86d810ab0cf34941a343eef1593632bf (table default.Table_9f2d3cc9c6724dab9ba35ff24e85c236 [id=c358e28088d44cf598c96a1bd2a4321e]). Expected version 1 got 0
I20240503 14:12:29.779592 28661 ts_tablet_manager.cc:1888] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:29.870117 28661 log.cc:1196] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/13c7a51c68ef478e9bdc5b3e3dc3d5df
I20240503 14:12:29.870158 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:29.870496 28661 ts_tablet_manager.cc:1909] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:29.870728 28029 catalog_manager.cc:5484] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:29.871563 28661 tablet_replica.cc:330] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:29.861598 28024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 114) took 1055 ms (client timeout 1000 ms). Trace:
I20240503 14:12:29.871811 28661 raft_consensus.cc:2229] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.871711 30446 consensus_queue.cc:1035] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:29.871814 28024 rpcz_store.cc:269] 0503 14:12:28.805711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.492037 (+686326us) service_pool.cc:225] Handling call
0503 14:12:29.861485 (+369448us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":21}
I20240503 14:12:29.794994 30335 raft_consensus.cc:482] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.802099 29996 consensus_queue.cc:1035] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.074s
I20240503 14:12:29.876420 30335 raft_consensus.cc:504] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.820106 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.876999 30335 leader_election.cc:288] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.789418 28604 tablet_bootstrap.cc:654] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:29.877454 28150 leader_election.cc:341] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.877629 28150 leader_election.cc:302] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.878177 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.878417 28212 raft_consensus.cc:2454] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:29.878870 30266 raft_consensus.cc:2726] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.878906 28427 leader_election.cc:302] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.879165 30335 raft_consensus.cc:2781] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.879236 30335 raft_consensus.cc:482] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.879307 30335 raft_consensus.cc:3037] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.881691 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.882365 30335 raft_consensus.cc:504] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:29.882493 28427 leader_election.cc:341] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.883055 30335 leader_election.cc:288] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.879058 28029 catalog_manager.cc:5484] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.883685 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:29.883903 28212 raft_consensus.cc:3037] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.884205 28029 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3e7b2865e9604ce58280321c7ee2a88e (table default.Table_ca6cee392d934dd3ac773e100ae031d2 [id=32be054076954aed9265888ca62e510b]). Expected version 4 got 0
I20240503 14:12:29.887022 28212 raft_consensus.cc:2454] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:29.887825 28427 leader_election.cc:302] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:29.888214 30335 raft_consensus.cc:2781] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:29.888782 30335 raft_consensus.cc:686] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:29.889113 30335 consensus_queue.cc:237] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.889467 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:29.891662 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 0f642b306a8c49fd9c070a9d340889a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f642b306a8c49fd9c070a9d340889a2 already in progress: creating tablet
I20240503 14:12:29.893196 28661 raft_consensus.cc:2258] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.894397 30404 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 1051) took 1011 ms. Trace:
I20240503 14:12:29.894506 30404 rpcz_store.cc:276] 0503 14:12:28.883007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.883081 (+ 74us) service_pool.cc:225] Handling call
0503 14:12:29.679775 (+796694us) ts_tablet_manager.cc:1120] Deleting tablet 007a892ef8504b848a400c05740eaa0c
0503 14:12:29.679782 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.761713 (+ 81931us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.843581 (+ 81868us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.894366 (+ 50785us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":796498}
I20240503 14:12:29.894675 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 2c99f41d735e4a92bda35ff68dca77c9 (DEFAULT_TABLE table=default.Table_a7d0ab9ea1144a03899dd81992e69db2 [id=773b5174b5064c3ba3e7683008517cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.894942 30404 tablet_replica.cc:330] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:29.895144 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c99f41d735e4a92bda35ff68dca77c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.895153 30404 raft_consensus.cc:2229] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.895318 30404 raft_consensus.cc:2258] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.895926 30404 ts_tablet_manager.cc:1875] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.895993 28600 ts_tablet_manager.cc:1398] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.047s user 0.003s sys 0.000s
I20240503 14:12:29.896481 30263 raft_consensus.cc:482] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.896629 30263 raft_consensus.cc:504] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.897356 30263 leader_election.cc:288] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.899230 28600 tablet_bootstrap.cc:492] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:29.903609 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
W20240503 14:12:29.893756 28427 leader_election.cc:341] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.901403 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d810ab0cf34941a343eef1593632bf" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.907451 28290 leader_election.cc:302] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.907888 30263 raft_consensus.cc:2726] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.908691 28661 ts_tablet_manager.cc:1875] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.911597 28600 tablet_bootstrap.cc:654] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.912858 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 2c99f41d735e4a92bda35ff68dca77c9 (DEFAULT_TABLE table=default.Table_a7d0ab9ea1144a03899dd81992e69db2 [id=773b5174b5064c3ba3e7683008517cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.913336 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c99f41d735e4a92bda35ff68dca77c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.915010 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 1712b3f636e64d19b8f9d00c5d564f55 (DEFAULT_TABLE table=default.Table_ccd689a56b554370857c1833876a7851 [id=17ca6c4f218a477a89a77c319189d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.915480 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1712b3f636e64d19b8f9d00c5d564f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.916721 30263 raft_consensus.cc:482] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.916874 30263 raft_consensus.cc:504] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:29.917536 30263 leader_election.cc:288] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.918300 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:29.925374 28614 tablet_bootstrap.cc:492] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:29.926970 28614 ts_tablet_manager.cc:1367] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.072s user 0.019s sys 0.006s
I20240503 14:12:29.928380 28614 raft_consensus.cc:348] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.930984 28614 raft_consensus.cc:374] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:29.931792 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 1712b3f636e64d19b8f9d00c5d564f55 (DEFAULT_TABLE table=default.Table_ccd689a56b554370857c1833876a7851 [id=17ca6c4f218a477a89a77c319189d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.932402 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1712b3f636e64d19b8f9d00c5d564f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.913231 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 2c99f41d735e4a92bda35ff68dca77c9 (DEFAULT_TABLE table=default.Table_a7d0ab9ea1144a03899dd81992e69db2 [id=773b5174b5064c3ba3e7683008517cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.935370 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c99f41d735e4a92bda35ff68dca77c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.933549 28318 tablet_service.cc:1467] Processing CreateTablet for tablet ee716bdc671940fe92547a5c94972d4f (DEFAULT_TABLE table=default.Table_84bf4809af6d4c26bdb71f0ae680cfb6 [id=9ff470c2d3364ea6b0eb9dcd469a750f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.939944 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee716bdc671940fe92547a5c94972d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.921350 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acbc154c03c47c1b2f9f433fcb2ef9a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.944702 28207 raft_consensus.cc:2379] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:29.945617 28290 leader_election.cc:302] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:29.946197 30263 raft_consensus.cc:3037] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.949133 30263 raft_consensus.cc:2726] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:29.949379 28196 tablet_service.cc:1467] Processing CreateTablet for tablet ee716bdc671940fe92547a5c94972d4f (DEFAULT_TABLE table=default.Table_84bf4809af6d4c26bdb71f0ae680cfb6 [id=9ff470c2d3364ea6b0eb9dcd469a750f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.950012 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee716bdc671940fe92547a5c94972d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.950556 28614 raft_consensus.cc:729] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.951406 28344 raft_consensus.cc:3037] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.951794 28614 consensus_queue.cc:260] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.956976 28614 ts_tablet_manager.cc:1398] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240503 14:12:29.957402 28614 tablet_bootstrap.cc:492] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:29.957572 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 014302d49cd94751b3af961d49f4fcc6 (DEFAULT_TABLE table=default.Table_b3d87c07d8a8429693540260e15e7362 [id=e96fe8bab30e4a61b0278f7a398a546c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.958027 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014302d49cd94751b3af961d49f4fcc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.958500 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb570eb39d54f2cac41d4f77b18abc7 (DEFAULT_TABLE table=default.Table_8ce861018efe45e58f9c7ed20c85a6d7 [id=787502bba02a48bba7f368d83f640a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.958860 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb570eb39d54f2cac41d4f77b18abc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.958962 28614 tablet_bootstrap.cc:654] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:29.960067 28466 tablet_service.cc:1467] Processing CreateTablet for tablet ee716bdc671940fe92547a5c94972d4f (DEFAULT_TABLE table=default.Table_84bf4809af6d4c26bdb71f0ae680cfb6 [id=9ff470c2d3364ea6b0eb9dcd469a750f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.960552 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee716bdc671940fe92547a5c94972d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.965608 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 014302d49cd94751b3af961d49f4fcc6 (DEFAULT_TABLE table=default.Table_b3d87c07d8a8429693540260e15e7362 [id=e96fe8bab30e4a61b0278f7a398a546c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.966100 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014302d49cd94751b3af961d49f4fcc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.968209 30335 consensus_queue.cc:1035] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:29.968425 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb570eb39d54f2cac41d4f77b18abc7 (DEFAULT_TABLE table=default.Table_8ce861018efe45e58f9c7ed20c85a6d7 [id=787502bba02a48bba7f368d83f640a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.968878 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb570eb39d54f2cac41d4f77b18abc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.969206 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e8411f474c247c2abfd0bd9682fb8a2 with delete_type TABLET_DATA_DELETED (replaced by tablet a1ebec4622ef4950bd1c70451958c59b) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.969270 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb570eb39d54f2cac41d4f77b18abc7 (DEFAULT_TABLE table=default.Table_8ce861018efe45e58f9c7ed20c85a6d7 [id=787502bba02a48bba7f368d83f640a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.969478 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d45788ae644a8581acca54cc55dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.969646 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb570eb39d54f2cac41d4f77b18abc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.969727 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.969897 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.970137 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d54cc5c03034a6685c1da34f9bd65ac with delete_type TABLET_DATA_DELETED (replaced by tablet d7c79ffee3954b5c870f4a8aa6379517) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.970242 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet c9325d5142af4b018354bf38370b61e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 188d7b644d2842109f1e0c124903ddd9) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.970414 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d1314841574fc8bcf3ccb16755e421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:20 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.970533 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8097ba5606cf42d68448833de36d85c7) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.970695 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4846d662a64d51b16fd5447c371e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.970929 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcc6cef7b7d464a9fbbae61caf0437d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.971066 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet e0175b38c1e0464a9a6208cfaf68ca0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.972210 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet a884cf4ea11e48ef941722b79ad50075 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.972417 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b176a071ba4f3fbff2a64d6e46027a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.972580 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b41e3b21347b393497de4d4e87469 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.976290 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df with delete_type TABLET_DATA_DELETED (replaced by tablet ae909ab6263a401490952e3f68533741) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:29.977649 29828 tablet_replica.cc:330] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:29.977955 29828 raft_consensus.cc:2229] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.978179 29828 raft_consensus.cc:2258] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.978857 29828 ts_tablet_manager.cc:1875] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:29.979202 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 014302d49cd94751b3af961d49f4fcc6 (DEFAULT_TABLE table=default.Table_b3d87c07d8a8429693540260e15e7362 [id=e96fe8bab30e4a61b0278f7a398a546c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.979563 30451 consensus_queue.cc:1035] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:29.979707 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014302d49cd94751b3af961d49f4fcc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.982573 30335 raft_consensus.cc:482] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.983008 30335 raft_consensus.cc:504] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.983805 30335 leader_election.cc:288] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:29.984512 30272 raft_consensus.cc:482] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:29.984580 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:29.932343 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 1712b3f636e64d19b8f9d00c5d564f55 (DEFAULT_TABLE table=default.Table_ccd689a56b554370857c1833876a7851 [id=17ca6c4f218a477a89a77c319189d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:29.985164 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1712b3f636e64d19b8f9d00c5d564f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.988473 28600 tablet_bootstrap.cc:492] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:29.988392 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:29.988632 28600 ts_tablet_manager.cc:1367] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.004s
I20240503 14:12:29.984690 30272 raft_consensus.cc:504] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.989430 28600 raft_consensus.cc:348] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.989645 30272 leader_election.cc:288] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:29.990799 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7b805aee554f8fa4c198ce3409c6e6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:29.991612 28150 leader_election.cc:341] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.991978 28212 raft_consensus.cc:2454] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:29.994506 28427 leader_election.cc:341] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:29.994855 28427 leader_election.cc:302] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:29.995182 30451 raft_consensus.cc:2781] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:29.995265 30451 raft_consensus.cc:482] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:29.995337 30451 raft_consensus.cc:3037] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:29.998499 30451 raft_consensus.cc:504] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:29.999230 30451 leader_election.cc:288] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.002249 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 48e7c8ead30343c0921ad9bab83d88d4 (DEFAULT_TABLE table=default.Table_33246ca4181441eaa0e703cfd07c1b3e [id=b401a6f165cd4b3493eebd2b6cfe646b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.002269 28197 tablet_service.cc:1467] Processing CreateTablet for tablet f79f6ee4b27c4685989d4636bdda6b66 (DEFAULT_TABLE table=default.Table_ae0e06cb36cd42238ccf6efe518640c2 [id=20447b89a68f454f8a8374ccb9f8d738]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.002837 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e7c8ead30343c0921ad9bab83d88d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.003038 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79f6ee4b27c4685989d4636bdda6b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.004410 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 5eaef52887064ff28c5d0fbbb4090d8f (DEFAULT_TABLE table=default.Table_8a699c7b01a04e4aad97227bb13be53a [id=59574dd4847c451b8de26e63f9924438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.004802 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eaef52887064ff28c5d0fbbb4090d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.008131 29828 ts_tablet_manager.cc:1888] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.008302 29828 log.cc:1196] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/02d45788ae644a8581acca54cc55dd3f
I20240503 14:12:30.008616 29828 ts_tablet_manager.cc:1909] T 02d45788ae644a8581acca54cc55dd3f P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.009167 29828 tablet_replica.cc:330] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.009407 29828 raft_consensus.cc:2229] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.009595 29828 raft_consensus.cc:2258] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:30.013267 27856 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 51192.7us
I20240503 14:12:30.014973 28604 tablet_bootstrap.cc:492] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:30.015146 28604 ts_tablet_manager.cc:1367] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.292s user 0.000s sys 0.007s
I20240503 14:12:30.016795 29828 ts_tablet_manager.cc:1875] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.018388 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:30.018708 28213 raft_consensus.cc:3037] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.019798 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df (table default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]) successfully deleted
I20240503 14:12:30.020032 28188 tablet_service.cc:1467] Processing CreateTablet for tablet a83532bf2852473a9c51ce7cc78b5d34 (DEFAULT_TABLE table=default.Table_a7060c6b06e14403ad4ab81fc6e5c8cd [id=ffbda772b5ce4606914f5fecfff4a335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.020524 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83532bf2852473a9c51ce7cc78b5d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.932014 30404 ts_tablet_manager.cc:1888] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.021989 30404 log.cc:1196] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6de9e159d2ba47039dd6bd207a9cddeb
I20240503 14:12:30.022871 28031 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1242) took 1044 ms. Trace:
I20240503 14:12:30.023036 28031 rpcz_store.cc:276] 0503 14:12:28.978263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.819000 (+840737us) service_pool.cc:225] Handling call
0503 14:12:29.820050 (+ 1050us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:29.820565 (+ 515us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:30.022530 (+201965us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:30.022667 (+ 137us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:30.022803 (+ 136us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:29.821278 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:29.821382 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:29.821390 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:29.821393 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:29.821494 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:29.821506 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:29.821511 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:29.821591 (+ 80us) tablet.cc:699] Row locks acquired
0503 14:12:29.821593 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:29.821636 (+ 43us) write_op.cc:270] Start()
0503 14:12:29.821674 (+ 38us) write_op.cc:276] Timestamp: P: 1714745549821599 usec, L: 0
0503 14:12:29.821678 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:29.822066 (+ 388us) log.cc:842] Serialized 433 byte log entry
0503 14:12:30.021168 (+199102us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.021958 (+ 790us) write_op.cc:301] APPLY: starting
0503 14:12:30.021972 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.022014 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.022039 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.022158 (+ 119us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.022191 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:30.022199 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:30.022271 (+ 72us) log.cc:842] Serialized 29 byte log entry
0503 14:12:30.022359 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.022378 (+ 19us) write_op.cc:454] Released schema lock
0503 14:12:30.022385 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.022437 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.022440 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":6656,"child_traces":[["op",{"apply.queue_time_us":755,"apply.run_cpu_time_us":533,"apply.run_wall_time_us":532,"num_ops":2,"prepare.queue_time_us":155,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":880,"replication_time_us":199475}]]}
W20240503 14:12:30.023893 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6cf0a1f98bf14aa48fc5b5a805716309 consensus state has no leader
I20240503 14:12:30.024541 28027 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1241) took 1055 ms. Trace:
I20240503 14:12:30.024842 28032 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1240) took 1071 ms. Trace:
I20240503 14:12:30.024962 28032 rpcz_store.cc:276] 0503 14:12:28.952849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.816550 (+863701us) service_pool.cc:225] Handling call
0503 14:12:29.817029 (+ 479us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:29.817237 (+ 208us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:29.817441 (+ 204us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:29.817529 (+ 88us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:30.023601 (+206072us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:30.024063 (+ 462us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:29.818138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:29.818244 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:29.818251 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:29.818253 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:29.818379 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:29.818392 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:29.818397 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:29.818466 (+ 69us) tablet.cc:699] Row locks acquired
0503 14:12:29.818469 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:29.818511 (+ 42us) write_op.cc:270] Start()
0503 14:12:29.818551 (+ 40us) write_op.cc:276] Timestamp: P: 1714745549818475 usec, L: 0
0503 14:12:29.818555 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:29.818750 (+ 195us) log.cc:842] Serialized 405 byte log entry
0503 14:12:30.021090 (+202340us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.021337 (+ 247us) write_op.cc:301] APPLY: starting
0503 14:12:30.021358 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.021406 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.021437 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.021498 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.021535 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:30.021543 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:30.021640 (+ 97us) log.cc:842] Serialized 23 byte log entry
0503 14:12:30.021750 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.021775 (+ 25us) write_op.cc:454] Released schema lock
0503 14:12:30.021783 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.021852 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.021856 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":194,"apply.run_cpu_time_us":578,"apply.run_wall_time_us":576,"num_ops":1,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":696,"replication_time_us":202500}]]}
I20240503 14:12:30.025689 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet cace51f8e5ff49f596c94f3698fdf9ef from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:30.026314 28032 master_service.cc:946] ReplaceTablet: received request to replace tablet 34238c9859724d97a32f2a1d9315edb1 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:30.026669 28027 rpcz_store.cc:276] 0503 14:12:28.968712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.818567 (+849855us) service_pool.cc:225] Handling call
0503 14:12:29.823271 (+ 4704us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:29.823777 (+ 506us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:30.024270 (+200493us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:30.024382 (+ 112us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:30.024472 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:29.825261 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:29.825354 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:29.825358 (+ 4us) write_op.cc:435] Acquired schema lock
0503 14:12:29.825360 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:29.825465 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:29.825475 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:29.825480 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:29.825550 (+ 70us) tablet.cc:699] Row locks acquired
0503 14:12:29.825552 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:29.825589 (+ 37us) write_op.cc:270] Start()
0503 14:12:29.825612 (+ 23us) write_op.cc:276] Timestamp: P: 1714745549825553 usec, L: 0
0503 14:12:29.825616 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:29.825765 (+ 149us) log.cc:842] Serialized 433 byte log entry
0503 14:12:30.021247 (+195482us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.023006 (+ 1759us) write_op.cc:301] APPLY: starting
0503 14:12:30.023023 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.023055 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.023077 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.023222 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.023252 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:30.023260 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:30.023316 (+ 56us) log.cc:842] Serialized 29 byte log entry
0503 14:12:30.023360 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.023380 (+ 20us) write_op.cc:454] Released schema lock
0503 14:12:30.023385 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.023442 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.023446 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1389952,"child_traces":[["op",{"apply.queue_time_us":1725,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":487,"num_ops":2,"prepare.queue_time_us":966,"prepare.run_cpu_time_us":554,"prepare.run_wall_time_us":554,"replication_time_us":195618}]]}
I20240503 14:12:30.027482 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_78d5622f17704a729d8fe709d43e07c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.028203 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78d5622f17704a729d8fe709d43e07c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.028582 28197 tablet_service.cc:1467] Processing CreateTablet for tablet f2c56d8767cb4217a4a9b2e8be2e4e96 (DEFAULT_TABLE table=default.Table_b554e9ad820f41199f4ee1e53469d221 [id=faf2ed829d8f40dfa9dc368b74f0dae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.029114 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c56d8767cb4217a4a9b2e8be2e4e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.023897 28031 master_service.cc:946] ReplaceTablet: received request to replace tablet d9c7b5fb9b4a483c8292f68387591034 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:30.032572 30404 ts_tablet_manager.cc:1909] T 6de9e159d2ba47039dd6bd207a9cddeb P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:30.035382 28213 raft_consensus.cc:2454] T 86f1ccc26f50432c9ef8789c01ef3519 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:30.038347 30404 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58172 (request call id 1052) took 1154 ms. Trace:
I20240503 14:12:30.035382 29996 raft_consensus.cc:482] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.038476 30404 rpcz_store.cc:276] 0503 14:12:28.883834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:28.883878 (+ 44us) service_pool.cc:225] Handling call
0503 14:12:29.894882 (+1011004us) ts_tablet_manager.cc:1120] Deleting tablet 6de9e159d2ba47039dd6bd207a9cddeb
0503 14:12:29.894895 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:29.923207 (+ 28312us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:29.931883 (+ 8676us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:30.038317 (+106434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1010839}
I20240503 14:12:30.038750 29996 raft_consensus.cc:504] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.039467 29996 leader_election.cc:288] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.015962 28604 raft_consensus.cc:348] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.039822 28604 raft_consensus.cc:374] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.039891 28604 raft_consensus.cc:729] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.040191 28604 consensus_queue.cc:260] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.042590 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7779f7a9544e17aea7135eab2ccd5f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.042886 28213 raft_consensus.cc:2454] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.043376 28604 ts_tablet_manager.cc:1398] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240503 14:12:30.043721 28604 tablet_bootstrap.cc:492] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:30.044139 29828 ts_tablet_manager.cc:1888] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.044293 29828 log.cc:1196] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/2cad43f3c6094253ba87d0ed3fc29fd9
I20240503 14:12:30.044607 29828 ts_tablet_manager.cc:1909] T 2cad43f3c6094253ba87d0ed3fc29fd9 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.045276 30272 raft_consensus.cc:482] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.045429 30272 raft_consensus.cc:504] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.045933 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 48e7c8ead30343c0921ad9bab83d88d4 (DEFAULT_TABLE table=default.Table_33246ca4181441eaa0e703cfd07c1b3e [id=b401a6f165cd4b3493eebd2b6cfe646b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.046068 30272 leader_election.cc:288] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.046344 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e7c8ead30343c0921ad9bab83d88d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.047951 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.048445 28459 tablet_service.cc:1467] Processing CreateTablet for tablet f79f6ee4b27c4685989d4636bdda6b66 (DEFAULT_TABLE table=default.Table_ae0e06cb36cd42238ccf6efe518640c2 [id=20447b89a68f454f8a8374ccb9f8d738]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:30.048624 28150 leader_election.cc:341] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.048841 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79f6ee4b27c4685989d4636bdda6b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.049772 28604 tablet_bootstrap.cc:654] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.051802 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:30.052734 29828 tablet_replica.cc:330] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.052960 29828 raft_consensus.cc:2229] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:30.053850 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7779f7a9544e17aea7135eab2ccd5f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.054060 28344 raft_consensus.cc:2454] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.054924 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 48e7c8ead30343c0921ad9bab83d88d4 (DEFAULT_TABLE table=default.Table_33246ca4181441eaa0e703cfd07c1b3e [id=b401a6f165cd4b3493eebd2b6cfe646b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.055330 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e7c8ead30343c0921ad9bab83d88d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.055660 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 5eaef52887064ff28c5d0fbbb4090d8f (DEFAULT_TABLE table=default.Table_8a699c7b01a04e4aad97227bb13be53a [id=59574dd4847c451b8de26e63f9924438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.056107 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eaef52887064ff28c5d0fbbb4090d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.058383 28326 tablet_service.cc:1467] Processing CreateTablet for tablet f79f6ee4b27c4685989d4636bdda6b66 (DEFAULT_TABLE table=default.Table_ae0e06cb36cd42238ccf6efe518640c2 [id=20447b89a68f454f8a8374ccb9f8d738]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.058763 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79f6ee4b27c4685989d4636bdda6b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.060189 28335 tablet_service.cc:1467] Processing CreateTablet for tablet a83532bf2852473a9c51ce7cc78b5d34 (DEFAULT_TABLE table=default.Table_a7060c6b06e14403ad4ab81fc6e5c8cd [id=ffbda772b5ce4606914f5fecfff4a335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.060562 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83532bf2852473a9c51ce7cc78b5d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.062116 28322 tablet_service.cc:1467] Processing CreateTablet for tablet f2c56d8767cb4217a4a9b2e8be2e4e96 (DEFAULT_TABLE table=default.Table_b554e9ad820f41199f4ee1e53469d221 [id=faf2ed829d8f40dfa9dc368b74f0dae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.062494 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c56d8767cb4217a4a9b2e8be2e4e96. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:30.063747 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 4e8411f474c247c2abfd0bd9682fb8a2 because the tablet was not found. No further retry: Not found: Tablet not found: 4e8411f474c247c2abfd0bd9682fb8a2
I20240503 14:12:30.064152 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 02d45788ae644a8581acca54cc55dd3f (table default.Table_94abbca6df6f456fb210dd71f21384af [id=7a0c738a91944ff192451d5b248df415]) successfully deleted
I20240503 14:12:30.064406 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 2cad43f3c6094253ba87d0ed3fc29fd9 (table default.Table_a9e8bd96c6824e9c8c64036f0848c876 [id=9e438ac1c3254c7aae6f81c707f433d7]) successfully deleted
I20240503 14:12:30.064589 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 007a892ef8504b848a400c05740eaa0c (table default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]) successfully deleted
I20240503 14:12:30.065200 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6de9e159d2ba47039dd6bd207a9cddeb (table default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a [id=f41fc50b1ca142f290aabb60f6cf0cf8]) successfully deleted
W20240503 14:12:30.065531 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 09f1f5216cc940888803ce8f6acd2a47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09f1f5216cc940888803ce8f6acd2a47 already in progress: creating tablet
W20240503 14:12:30.065853 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 8cd037e135904e2cab09cbf39fed0b13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cd037e135904e2cab09cbf39fed0b13 already in progress: creating tablet
W20240503 14:12:30.066083 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9a1bc119df7d4694bbe926611ef73245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a1bc119df7d4694bbe926611ef73245 already in progress: creating tablet
W20240503 14:12:30.066335 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 0f642b306a8c49fd9c070a9d340889a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f642b306a8c49fd9c070a9d340889a2 already in progress: creating tablet
I20240503 14:12:29.989661 28600 raft_consensus.cc:374] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.072558 28600 raft_consensus.cc:729] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.072892 28600 consensus_queue.cc:260] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.074355 28600 ts_tablet_manager.cc:1398] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.086s user 0.002s sys 0.000s
I20240503 14:12:30.078696 29828 raft_consensus.cc:2258] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.080528 29828 ts_tablet_manager.cc:1875] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.083828 28614 tablet_bootstrap.cc:492] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:30.084015 28614 ts_tablet_manager.cc:1367] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.005s
I20240503 14:12:30.084980 28614 raft_consensus.cc:348] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.085196 28614 raft_consensus.cc:374] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.085250 28614 raft_consensus.cc:729] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.085577 28614 consensus_queue.cc:260] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.086385 28614 ts_tablet_manager.cc:1398] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:30.086724 28614 tablet_bootstrap.cc:492] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:30.087025 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7b805aee554f8fa4c198ce3409c6e6" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.087705 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.088295 28614 tablet_bootstrap.cc:654] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:30.089816 28150 leader_election.cc:341] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.089967 28150 leader_election.cc:302] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.090498 30272 raft_consensus.cc:2726] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:30.091095 28150 leader_election.cc:341] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.091230 28150 leader_election.cc:302] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.091727 30272 raft_consensus.cc:2726] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:30.095183 28150 consensus_peers.cc:485] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:30.097352 28150 consensus_peers.cc:485] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.100127 29828 ts_tablet_manager.cc:1888] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.100339 29828 log.cc:1196] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:30.100736 29828 ts_tablet_manager.cc:1909] T 62e4f6e3908e420e9fa156ee24b437d6 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.052423 28461 tablet_service.cc:1467] Processing CreateTablet for tablet f2c56d8767cb4217a4a9b2e8be2e4e96 (DEFAULT_TABLE table=default.Table_b554e9ad820f41199f4ee1e53469d221 [id=faf2ed829d8f40dfa9dc368b74f0dae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.101701 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c56d8767cb4217a4a9b2e8be2e4e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.102531 29828 tablet_replica.cc:330] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.102663 28600 tablet_bootstrap.cc:492] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.102756 29828 raft_consensus.cc:2229] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.102911 29828 raft_consensus.cc:2258] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.103464 29828 ts_tablet_manager.cc:1875] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.052829 28469 tablet_service.cc:1467] Processing CreateTablet for tablet a83532bf2852473a9c51ce7cc78b5d34 (DEFAULT_TABLE table=default.Table_a7060c6b06e14403ad4ab81fc6e5c8cd [id=ffbda772b5ce4606914f5fecfff4a335]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.054121 28427 leader_election.cc:302] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:30.104267 28600 tablet_bootstrap.cc:654] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.073735 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 5eaef52887064ff28c5d0fbbb4090d8f (DEFAULT_TABLE table=default.Table_8a699c7b01a04e4aad97227bb13be53a [id=59574dd4847c451b8de26e63f9924438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.104471 29996 raft_consensus.cc:2781] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:30.104523 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83532bf2852473a9c51ce7cc78b5d34. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:30.104481 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 50296.9us
I20240503 14:12:30.105051 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eaef52887064ff28c5d0fbbb4090d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.105362 28427 leader_election.cc:302] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:30.107998 30446 consensus_queue.cc:1035] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.108636 30446 raft_consensus.cc:2781] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:30.108785 30446 raft_consensus.cc:482] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:12:30.108896 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 310abf35ae02497c9d3696a0c49774ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 310abf35ae02497c9d3696a0c49774ae already in progress: creating tablet
W20240503 14:12:30.109391 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet eff8442b6c02469eaedb08c51200829b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff8442b6c02469eaedb08c51200829b already in progress: creating tablet
W20240503 14:12:30.109668 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet dc31b2e907b04ea19a6f9f7fd8f57843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc31b2e907b04ea19a6f9f7fd8f57843 already in progress: creating tablet
I20240503 14:12:30.104707 29996 raft_consensus.cc:686] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:30.114548 29996 consensus_queue.cc:237] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.116216 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 62e4f6e3908e420e9fa156ee24b437d6 (table default.Table_5654996201694c1b90fc6e728e068ca3 [id=9a6689ece91646069be1867e8f141392]) successfully deleted
W20240503 14:12:30.116573 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 3d54cc5c03034a6685c1da34f9bd65ac because the tablet was not found. No further retry: Not found: Tablet not found: 3d54cc5c03034a6685c1da34f9bd65ac
W20240503 14:12:30.116864 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet c9325d5142af4b018354bf38370b61e0 because the tablet was not found. No further retry: Not found: Tablet not found: c9325d5142af4b018354bf38370b61e0
I20240503 14:12:30.117537 28026 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1244) took 1138 ms. Trace:
I20240503 14:12:30.117697 28026 rpcz_store.cc:276] 0503 14:12:28.979355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.828538 (+849183us) service_pool.cc:225] Handling call
0503 14:12:29.829652 (+ 1114us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:29.830165 (+ 513us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:30.117224 (+287059us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:30.117365 (+ 141us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:30.117462 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:29.854081 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:29.854242 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:29.854250 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:29.854253 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:29.854363 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:29.854376 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:29.854380 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:29.854469 (+ 89us) tablet.cc:699] Row locks acquired
0503 14:12:29.854472 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:29.855579 (+ 1107us) write_op.cc:270] Start()
0503 14:12:29.855632 (+ 53us) write_op.cc:276] Timestamp: P: 1714745549855528 usec, L: 0
0503 14:12:29.855638 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 14:12:29.856757 (+ 1119us) log.cc:842] Serialized 433 byte log entry
0503 14:12:30.115018 (+258261us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.116722 (+ 1704us) write_op.cc:301] APPLY: starting
0503 14:12:30.116736 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.116790 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.116818 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.116956 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.116988 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:30.116995 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:30.117064 (+ 69us) log.cc:842] Serialized 29 byte log entry
0503 14:12:30.117116 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.117132 (+ 16us) write_op.cc:454] Released schema lock
0503 14:12:30.117139 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.117203 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.117207 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1669,"apply.run_cpu_time_us":534,"apply.run_wall_time_us":534,"num_ops":2,"prepare.queue_time_us":23387,"prepare.run_cpu_time_us":894,"prepare.run_wall_time_us":4982,"replication_time_us":259362}]]}
I20240503 14:12:30.118521 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_acbb6ef1f5a84c11847e5299afebb8b0" } modify_external_catalogs: true
I20240503 14:12:30.118772 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_acbb6ef1f5a84c11847e5299afebb8b0" } modify_external_catalogs: true
I20240503 14:12:30.119482 28028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33344 (request call id 1239) took 1176 ms. Trace:
I20240503 14:12:30.119578 28028 rpcz_store.cc:276] 0503 14:12:28.943465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.814139 (+870674us) service_pool.cc:225] Handling call
0503 14:12:29.815265 (+ 1126us) catalog_manager.cc:2072] Acquired catalog manager lock
0503 14:12:29.815782 (+ 517us) catalog_manager.cc:2118] Created new table and tablet info
0503 14:12:30.119203 (+303421us) catalog_manager.cc:2178] Wrote table and tablets to system table
0503 14:12:30.119318 (+ 115us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0503 14:12:30.119411 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:29.848977 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:29.849197 (+ 220us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:29.849205 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:29.849208 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:29.849325 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:29.849340 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:29.849345 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:29.849438 (+ 93us) tablet.cc:699] Row locks acquired
0503 14:12:29.849441 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:29.849491 (+ 50us) write_op.cc:270] Start()
0503 14:12:29.849534 (+ 43us) write_op.cc:276] Timestamp: P: 1714745549849449 usec, L: 0
0503 14:12:29.849539 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:29.849770 (+ 231us) log.cc:842] Serialized 433 byte log entry
0503 14:12:30.114936 (+265166us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.116004 (+ 1068us) write_op.cc:301] APPLY: starting
0503 14:12:30.116027 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.116106 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.116140 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.116278 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.116318 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:12:30.116326 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:30.116419 (+ 93us) log.cc:842] Serialized 29 byte log entry
0503 14:12:30.116508 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.116534 (+ 26us) write_op.cc:454] Released schema lock
0503 14:12:30.116546 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.116619 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.116623 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":991,"apply.run_cpu_time_us":689,"apply.run_wall_time_us":688,"num_ops":2,"prepare.queue_time_us":22995,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":5050,"replication_time_us":265361}]]}
I20240503 14:12:30.120293 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1df0f31c455445fbac525618cff09685" } modify_external_catalogs: true
I20240503 14:12:30.120472 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1df0f31c455445fbac525618cff09685" } modify_external_catalogs: true
I20240503 14:12:30.124693 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b15ac98442fe4481b0495a342ae6a57a" } modify_external_catalogs: true
I20240503 14:12:30.124938 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b15ac98442fe4481b0495a342ae6a57a" } modify_external_catalogs: true
I20240503 14:12:30.128324 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2]). Expected version 3 got 0
I20240503 14:12:30.128751 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
I20240503 14:12:30.129125 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (table default.Table_bfff85704d88463383cbf94f9b121359 [id=bd1950deb4274d86bf288d501cf5c710]). Expected version 2 got 0
I20240503 14:12:30.129303 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 34238c9859724d97a32f2a1d9315edb1 (table default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]). Expected version 1 got 0
I20240503 14:12:30.129480 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:30.129611 28614 tablet_bootstrap.cc:492] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:30.129683 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:30.129787 28614 ts_tablet_manager.cc:1367] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.003s
I20240503 14:12:30.130234 28025 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
I20240503 14:12:30.130656 28614 raft_consensus.cc:348] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.130884 28614 raft_consensus.cc:374] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.130945 28614 raft_consensus.cc:729] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.133205 28600 tablet_bootstrap.cc:492] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.133598 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet d9c7b5fb9b4a483c8292f68387591034
I20240503 14:12:30.134263 28031 catalog_manager.cc:6554] ReplaceTablet: tablet d9c7b5fb9b4a483c8292f68387591034 deleted and replaced by tablet 6a9f6a750b744433b01f821506767a02
I20240503 14:12:30.134444 28031 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1245) took 1038 ms. Trace:
I20240503 14:12:30.134550 28031 rpcz_store.cc:276] 0503 14:12:29.095640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.023796 (+928156us) service_pool.cc:225] Handling call
0503 14:12:30.134361 (+110565us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:30.031349 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:30.031466 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:30.031473 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:30.031475 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:30.031592 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:30.031603 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:30.031608 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:30.031692 (+ 84us) tablet.cc:699] Row locks acquired
0503 14:12:30.031695 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:30.031736 (+ 41us) write_op.cc:270] Start()
0503 14:12:30.031771 (+ 35us) write_op.cc:276] Timestamp: P: 1714745550031683 usec, L: 0
0503 14:12:30.031775 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:30.031950 (+ 175us) log.cc:842] Serialized 628 byte log entry
0503 14:12:30.132384 (+100434us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.132662 (+ 278us) write_op.cc:301] APPLY: starting
0503 14:12:30.132681 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.132729 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.132756 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.132867 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.132923 (+ 56us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:30.132930 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:30.133016 (+ 86us) log.cc:842] Serialized 29 byte log entry
0503 14:12:30.133075 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.133094 (+ 19us) write_op.cc:454] Released schema lock
0503 14:12:30.133102 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.133166 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.133170 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":218,"apply.run_cpu_time_us":598,"apply.run_wall_time_us":598,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":699,"prepare.run_wall_time_us":699,"raft.queue_time_us":467,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":100576}]]}
I20240503 14:12:30.135978 28600 ts_tablet_manager.cc:1367] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.004s
I20240503 14:12:30.136293 28614 consensus_queue.cc:260] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.136546 30266 consensus_queue.cc:1035] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.138267 28614 ts_tablet_manager.cc:1398] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:12:30.138549 28600 raft_consensus.cc:348] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:30.140347 28427 leader_election.cc:341] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.140816 28614 tablet_bootstrap.cc:492] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
W20240503 14:12:30.140923 28427 consensus_peers.cc:485] T ac5189b6cdaa486c8c2f1b5c141d513b P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.140978 28600 raft_consensus.cc:374] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.141175 28600 raft_consensus.cc:729] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.141503 29996 consensus_queue.cc:1035] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.142004 28600 consensus_queue.cc:260] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.143517 28614 tablet_bootstrap.cc:654] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.144016 28600 ts_tablet_manager.cc:1398] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:30.144855 28600 tablet_bootstrap.cc:492] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.146864 28600 tablet_bootstrap.cc:654] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.151728 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
I20240503 14:12:30.153426 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (table default.Table_bfff85704d88463383cbf94f9b121359 [id=bd1950deb4274d86bf288d501cf5c710]). Expected version 2 got 0
I20240503 14:12:30.153945 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
W20240503 14:12:30.155719 28150 consensus_peers.cc:485] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.156370 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2]). Expected version 3 got 0
I20240503 14:12:30.156716 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:30.157447 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 86d810ab0cf34941a343eef1593632bf (table default.Table_9f2d3cc9c6724dab9ba35ff24e85c236 [id=c358e28088d44cf598c96a1bd2a4321e]). Expected version 1 got 0
I20240503 14:12:30.157843 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:30.158133 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 3e7b2865e9604ce58280321c7ee2a88e (table default.Table_ca6cee392d934dd3ac773e100ae031d2 [id=32be054076954aed9265888ca62e510b]). Expected version 4 got 0
I20240503 14:12:30.096189 28661 ts_tablet_manager.cc:1888] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.165056 28600 tablet_bootstrap.cc:492] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.165145 28661 log.cc:1196] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/cf810e84367f420b928a171790849445
I20240503 14:12:30.165510 28661 ts_tablet_manager.cc:1909] T cf810e84367f420b928a171790849445 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:30.165746 28600 ts_tablet_manager.cc:1367] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240503 14:12:30.166863 28600 raft_consensus.cc:348] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.168280 28614 tablet_bootstrap.cc:492] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:30.168527 28614 ts_tablet_manager.cc:1367] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20240503 14:12:30.169520 28614 raft_consensus.cc:348] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.169834 28614 raft_consensus.cc:374] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.169945 28614 raft_consensus.cc:729] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.170222 28614 consensus_queue.cc:260] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.171012 28614 ts_tablet_manager.cc:1398] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:30.171176 28600 raft_consensus.cc:374] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.171586 28614 tablet_bootstrap.cc:492] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:30.171697 28600 raft_consensus.cc:729] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.172326 28661 tablet_replica.cc:330] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:30.172297 28600 consensus_queue.cc:260] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.173787 28661 raft_consensus.cc:2229] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:30.174602 28661 pending_rounds.cc:62] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:30.174917 28600 ts_tablet_manager.cc:1398] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:12:30.175433 28600 tablet_bootstrap.cc:492] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.177255 28661 pending_rounds.cc:68] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597770594459648 op_type: NO_OP noop_request { }
I20240503 14:12:30.177552 28661 raft_consensus.cc:2866] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:30.177711 28614 tablet_bootstrap.cc:654] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.177978 28661 raft_consensus.cc:2258] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.179677 28600 tablet_bootstrap.cc:654] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.179746 30315 raft_consensus.cc:482] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.179910 30315 raft_consensus.cc:504] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.180588 30315 leader_election.cc:288] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.182029 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49445f0f1dd1451c9ba066e61273f2f1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:30.186702 28025 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.193:45997 (request call id 105) took 1362 ms (client timeout 1000 ms). Trace:
W20240503 14:12:30.187008 28025 rpcz_store.cc:269] 0503 14:12:28.823775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.505511 (+681736us) service_pool.cc:225] Handling call
0503 14:12:30.186652 (+681141us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":29}
I20240503 14:12:30.189299 28614 tablet_bootstrap.cc:492] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:30.189555 28614 ts_tablet_manager.cc:1367] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20240503 14:12:30.190474 28614 raft_consensus.cc:348] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.190794 28614 raft_consensus.cc:374] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.190913 28614 raft_consensus.cc:729] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.191205 28614 consensus_queue.cc:260] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.186209 28604 tablet_bootstrap.cc:492] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:30.192409 29828 ts_tablet_manager.cc:1888] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.192524 29828 log.cc:1196] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a3d1314841574fc8bcf3ccb16755e421
I20240503 14:12:30.192817 29828 ts_tablet_manager.cc:1909] T a3d1314841574fc8bcf3ccb16755e421 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.192590 28604 ts_tablet_manager.cc:1367] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.149s user 0.002s sys 0.004s
W20240503 14:12:30.193236 28265 tablet.cc:2369] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:30.193800 28604 raft_consensus.cc:348] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.194001 28604 raft_consensus.cc:374] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.194049 28604 raft_consensus.cc:729] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.194442 28600 tablet_bootstrap.cc:492] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.194590 28600 ts_tablet_manager.cc:1367] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 14:12:30.195566 28600 raft_consensus.cc:348] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.195777 28600 raft_consensus.cc:374] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.195829 28600 raft_consensus.cc:729] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.196053 28600 consensus_queue.cc:260] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:30.186554 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 45162.2us
I20240503 14:12:30.196784 28600 ts_tablet_manager.cc:1398] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:30.197256 28600 tablet_bootstrap.cc:492] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.200501 28600 tablet_bootstrap.cc:654] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.200982 30451 raft_consensus.cc:482] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.201153 30451 raft_consensus.cc:504] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:30.201536 28290 leader_election.cc:341] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:30.201797 28290 leader_election.cc:341] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 49445f0f1dd1451c9ba066e61273f2f1
I20240503 14:12:30.201885 30451 leader_election.cc:288] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:30.201905 28290 leader_election.cc:302] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.202175 30315 raft_consensus.cc:2726] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.209547 30558 consensus_queue.cc:1035] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.210105 30558 consensus_queue.cc:1035] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.218294 28600 tablet_bootstrap.cc:492] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.218533 28600 ts_tablet_manager.cc:1367] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.002s
I20240503 14:12:30.219648 28600 raft_consensus.cc:348] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.220002 28600 raft_consensus.cc:374] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.220127 28600 raft_consensus.cc:729] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.220469 28600 consensus_queue.cc:260] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.222162 28600 ts_tablet_manager.cc:1398] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:30.227052 30315 raft_consensus.cc:482] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.227254 30315 raft_consensus.cc:504] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.227950 30315 leader_election.cc:288] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
W20240503 14:12:30.228530 28290 consensus_peers.cc:485] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 -> Peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Couldn't send request to peer d472d9f94d8949cf86284d32a161cebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13c7a51c68ef478e9bdc5b3e3dc3d5df. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.229734 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.230058 28207 raft_consensus.cc:2379] T b59c0bc115d949f69d80e1f3628d5a79 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:30.231024 28600 tablet_bootstrap.cc:492] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.234030 30558 raft_consensus.cc:482] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.234241 30558 raft_consensus.cc:504] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.234540 28600 tablet_bootstrap.cc:654] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.234908 30558 leader_election.cc:288] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:30.237565 30272 raft_consensus.cc:482] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.237738 30272 raft_consensus.cc:504] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.238381 30272 leader_election.cc:288] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.240564 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f5103b31ed45b480252dcf94b72f9c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.241945 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f5103b31ed45b480252dcf94b72f9c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:30.242741 28150 leader_election.cc:341] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:30.243409 28150 leader_election.cc:341] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.243530 28150 leader_election.cc:302] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:30.243584 28427 leader_election.cc:341] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b476e87faea1415d95ae014a93fa22af
I20240503 14:12:30.243782 30272 raft_consensus.cc:2726] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.249184 30272 raft_consensus.cc:482] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.249372 30272 raft_consensus.cc:504] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.250108 30272 leader_election.cc:288] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.251102 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7b6df20b14d1ead036cdc42174d3a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.251102 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.251333 28207 raft_consensus.cc:2454] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.251716 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7b6df20b14d1ead036cdc42174d3a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:30.252622 28150 leader_election.cc:341] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:30.252866 28150 leader_election.cc:341] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.252971 28150 leader_election.cc:302] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.253382 30272 raft_consensus.cc:2726] T 5ac7b6df20b14d1ead036cdc42174d3a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.256271 28427 leader_election.cc:302] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:30.256592 30558 raft_consensus.cc:2781] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:30.256664 30558 raft_consensus.cc:482] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:30.256724 30558 raft_consensus.cc:3037] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.259330 30558 raft_consensus.cc:504] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.261665 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:30.261932 28207 raft_consensus.cc:3037] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.264945 28207 raft_consensus.cc:2454] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:30.268266 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e87faea1415d95ae014a93fa22af" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.268563 28344 raft_consensus.cc:2454] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.269739 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.271158 29828 tablet_replica.cc:330] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.271428 29828 raft_consensus.cc:2229] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.271660 29828 raft_consensus.cc:2258] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.272321 29828 ts_tablet_manager.cc:1875] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.260056 30558 leader_election.cc:288] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:30.261123 30335 consensus_queue.cc:1035] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20240503 14:12:30.264683 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c0bc115d949f69d80e1f3628d5a79" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.188524 28661 ts_tablet_manager.cc:1875] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.283732 28604 consensus_queue.cc:260] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.288497 28600 tablet_bootstrap.cc:492] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.288668 28600 ts_tablet_manager.cc:1367] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.012s
I20240503 14:12:30.289602 28600 raft_consensus.cc:348] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.289803 28600 raft_consensus.cc:374] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.289841 28600 raft_consensus.cc:729] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.290041 28600 consensus_queue.cc:260] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.290551 28600 ts_tablet_manager.cc:1398] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:30.290539 30335 raft_consensus.cc:482] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a64bc15d65d84ad5a005960a9a84df9c)
I20240503 14:12:30.290688 30335 raft_consensus.cc:504] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.290843 28600 tablet_bootstrap.cc:492] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.291134 28604 ts_tablet_manager.cc:1398] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.098s user 0.002s sys 0.000s
I20240503 14:12:30.291325 30335 leader_election.cc:288] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.291580 28604 tablet_bootstrap.cc:492] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:30.294034 28600 tablet_bootstrap.cc:654] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:30.296367 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 164303us
I20240503 14:12:30.297096 30315 raft_consensus.cc:482] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.297271 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet d9c7b5fb9b4a483c8292f68387591034 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a9f6a750b744433b01f821506767a02) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.297252 30315 raft_consensus.cc:504] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:30.297909 28408 tablet.cc:2369] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:30.297972 30315 leader_election.cc:288] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.301153 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e87faea1415d95ae014a93fa22af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.304862 28290 leader_election.cc:302] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.307583 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.50.254:36571 every 8 connection(s)
I20240503 14:12:30.308880 30315 raft_consensus.cc:3037] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.308895 30103 rpc_server.cc:307] RPC server started. Bound to: 127.27.50.254:36571
I20240503 14:12:30.311828 30315 raft_consensus.cc:2726] T b59c0bc115d949f69d80e1f3628d5a79 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.312383 30103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/master-0/data/info.pb
I20240503 14:12:30.313215 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.314200 30446 raft_consensus.cc:3037] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.314824 28600 tablet_bootstrap.cc:492] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.315086 28600 ts_tablet_manager.cc:1367] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.006s
I20240503 14:12:30.315606 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.315842 28207 raft_consensus.cc:2454] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 2.
I20240503 14:12:30.316179 28600 raft_consensus.cc:348] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.316371 28600 raft_consensus.cc:374] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.316421 28600 raft_consensus.cc:729] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.317338 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet d9c7b5fb9b4a483c8292f68387591034 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a9f6a750b744433b01f821506767a02) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.317553 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e87faea1415d95ae014a93fa22af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.317734 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet a31555fc800142bd849dbd572a194002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.317566 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 646f83185d5241268d4126d10561e937 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.317894 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.317981 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.318048 28427 leader_election.cc:302] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:30.318118 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.318312 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.318435 30335 raft_consensus.cc:2781] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:30.318528 30335 raft_consensus.cc:482] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:30.318576 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:30.318655 28427 leader_election.cc:341] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.318593 30335 raft_consensus.cc:3037] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.319173 27851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmRrLf/build/asan/bin/kudu as pid 30103
I20240503 14:12:30.320286 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfd6d40a57e4bea973874ff12c0ea3a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.320524 28212 raft_consensus.cc:2379] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:30.192824 28614 ts_tablet_manager.cc:1398] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:30.321305 28614 tablet_bootstrap.cc:492] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:30.322558 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet d9c7b5fb9b4a483c8292f68387591034 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a9f6a750b744433b01f821506767a02) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.323534 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:30.324985 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.325222 28344 raft_consensus.cc:2379] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 2: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:30.316633 28600 consensus_queue.cc:260] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.327900 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_346e6ce2bc2b4524a52ec6ac63cbe9d8" } new_table_name: "default.Table_114d45df76c540f29b8f41b3739acd33" modify_external_catalogs: true
I20240503 14:12:30.328194 28600 ts_tablet_manager.cc:1398] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:30.328562 28600 tablet_bootstrap.cc:492] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.330289 28600 tablet_bootstrap.cc:654] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.331465 28290 leader_election.cc:302] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.332283 28031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7fb65f72628b40a4855474fe07921278" } modify_external_catalogs: true
I20240503 14:12:30.332523 28031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7fb65f72628b40a4855474fe07921278" } modify_external_catalogs: true
I20240503 14:12:30.333178 28025 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_461c1f89931e46ed9efa3024088d2adb" } modify_external_catalogs: true
I20240503 14:12:30.333403 28025 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_461c1f89931e46ed9efa3024088d2adb" } modify_external_catalogs: true
I20240503 14:12:30.334679 30315 raft_consensus.cc:3037] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.337330 30315 raft_consensus.cc:2726] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.340484 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet c816f3b05cfb4496984f6cb28d310cc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8097ba5606cf42d68448833de36d85c7) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.342092 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.345430 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.346525 28600 tablet_bootstrap.cc:492] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.346536 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.346707 28600 ts_tablet_manager.cc:1367] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 14:12:30.347499 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c0540960ac45aabeaf13f888caf238) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.347596 28600 raft_consensus.cc:348] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.347796 30404 tablet_replica.cc:330] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:30.348016 30404 raft_consensus.cc:2229] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.348390 30404 raft_consensus.cc:2258] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.348990 30404 ts_tablet_manager.cc:1875] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.347811 28600 raft_consensus.cc:374] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.349320 28600 raft_consensus.cc:729] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.349551 28600 consensus_queue.cc:260] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.350859 28600 ts_tablet_manager.cc:1398] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:30.351179 28600 tablet_bootstrap.cc:492] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.353189 28600 tablet_bootstrap.cc:654] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.355787 30404 ts_tablet_manager.cc:1888] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.356037 30404 log.cc:1196] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/d7c79ffee3954b5c870f4a8aa6379517
I20240503 14:12:30.356519 30404 ts_tablet_manager.cc:1909] T d7c79ffee3954b5c870f4a8aa6379517 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:30.358269 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.358600 30404 tablet_replica.cc:330] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:30.359378 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 646f83185d5241268d4126d10561e937 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.359926 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.360620 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.361042 28614 tablet_bootstrap.cc:654] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.362820 30404 raft_consensus.cc:2229] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.364038 30272 raft_consensus.cc:482] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.364240 30272 raft_consensus.cc:504] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.365151 30272 leader_election.cc:288] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.365919 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45df464f6954a56bec8b107b350320c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:30.366834 28150 leader_election.cc:341] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.367193 30404 pending_rounds.cc:62] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:30.367298 30404 pending_rounds.cc:68] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597769800589312 op_type: NO_OP noop_request { }
I20240503 14:12:30.367484 30404 raft_consensus.cc:2866] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:30.367642 30404 raft_consensus.cc:2258] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.368731 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.369571 30404 ts_tablet_manager.cc:1875] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.381032 28600 tablet_bootstrap.cc:492] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.381228 28600 ts_tablet_manager.cc:1367] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 14:12:30.382071 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.382231 28600 raft_consensus.cc:348] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.382448 28600 raft_consensus.cc:374] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.382515 28600 raft_consensus.cc:729] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.382854 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet a31555fc800142bd849dbd572a194002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.382828 28600 consensus_queue.cc:260] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.383431 28600 ts_tablet_manager.cc:1398] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:30.383363 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:30.319664 28427 consensus_peers.cc:485] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.321645 30335 raft_consensus.cc:504] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:30.383940 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 64309.6us
I20240503 14:12:30.384310 28427 leader_election.cc:302] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:30.384547 30335 leader_election.cc:288] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:30.384737 30335 raft_consensus.cc:2781] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:30.384857 30335 raft_consensus.cc:686] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:30.385115 30335 consensus_queue.cc:237] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.389132 28600 tablet_bootstrap.cc:492] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.332649 30446 raft_consensus.cc:504] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.397773 30446 leader_election.cc:288] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.327461 28604 tablet_bootstrap.cc:654] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.405753 30593 sys_catalog.cc:263] Verifying existing consensus state
I20240503 14:12:30.408897 28263 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:30.412521 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac2833a0bb045dcb2824468d936ffe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:11 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.412663 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.412865 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.412885 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d45788ae644a8581acca54cc55dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:09 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.413064 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.413236 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 49445f0f1dd1451c9ba066e61273f2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.413388 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e87faea1415d95ae014a93fa22af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.413518 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet af0137d7b1034a79bf4947c7d3f3fd57) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.413579 28600 tablet_bootstrap.cc:654] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.414876 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e14e8b5f4046029af4673a277112d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:23 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.417383 28403 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:30.428977 28600 tablet_bootstrap.cc:492] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.429252 28600 ts_tablet_manager.cc:1367] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.008s
I20240503 14:12:30.430397 28600 raft_consensus.cc:348] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.430740 28600 raft_consensus.cc:374] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.430845 28600 raft_consensus.cc:729] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.431138 28600 consensus_queue.cc:260] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:30.433221 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 48172.6us
I20240503 14:12:30.433562 28427 leader_election.cc:302] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:30.433908 30558 raft_consensus.cc:2781] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:30.434003 30558 raft_consensus.cc:482] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a64bc15d65d84ad5a005960a9a84df9c)
I20240503 14:12:30.434083 30558 raft_consensus.cc:3037] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:30.434510 28600 ts_tablet_manager.cc:1398] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:30.435532 30593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Bootstrap starting.
I20240503 14:12:30.436039 28600 tablet_bootstrap.cc:492] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.437121 30558 raft_consensus.cc:504] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.437870 30558 leader_election.cc:288] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.439399 28614 tablet_bootstrap.cc:492] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:30.439633 28614 ts_tablet_manager.cc:1367] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.118s user 0.002s sys 0.002s
I20240503 14:12:30.440608 28614 raft_consensus.cc:348] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.441251 28600 tablet_bootstrap.cc:654] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.366425 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45df464f6954a56bec8b107b350320c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.373826 30566 raft_consensus.cc:482] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.442472 30566 raft_consensus.cc:504] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.443265 30566 leader_election.cc:288] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:30.443346 28150 leader_election.cc:341] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.443475 28150 leader_election.cc:302] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.339840 28661 ts_tablet_manager.cc:1888] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.443805 28661 log.cc:1196] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/49445f0f1dd1451c9ba066e61273f2f1
I20240503 14:12:30.443773 30272 raft_consensus.cc:2726] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.444316 28661 ts_tablet_manager.cc:1909] T 49445f0f1dd1451c9ba066e61273f2f1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:30.445940 28661 tablet_replica.cc:330] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:30.454891 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a3d1314841574fc8bcf3ccb16755e421 (table default.Table_ddd5f026bb204de99edef3cc54bd8347 [id=e5c443cd73d340e99806c89d5b8e659f]) successfully deleted
I20240503 14:12:30.457311 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet cf810e84367f420b928a171790849445 (table default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]) successfully deleted
I20240503 14:12:30.469621 28614 raft_consensus.cc:374] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.469832 28614 raft_consensus.cc:729] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.470149 28614 consensus_queue.cc:260] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.470890 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7779f7a9544e17aea7135eab2ccd5f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:30.471171 28212 raft_consensus.cc:3037] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.474556 28600 tablet_bootstrap.cc:492] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.474726 28600 ts_tablet_manager.cc:1367] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.003s
I20240503 14:12:30.474941 28212 raft_consensus.cc:2454] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:30.475843 28600 raft_consensus.cc:348] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.476071 28600 raft_consensus.cc:374] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.476151 28600 raft_consensus.cc:729] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.476397 28600 consensus_queue.cc:260] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.477595 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:30.463181 30315 raft_consensus.cc:482] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.464176 28354 raft_consensus.cc:3037] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.488016 30315 raft_consensus.cc:504] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.488819 30315 leader_election.cc:288] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.469151 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e87faea1415d95ae014a93fa22af" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:30.491561 28344 raft_consensus.cc:3037] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.469445 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7779f7a9544e17aea7135eab2ccd5f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:30.492189 28349 raft_consensus.cc:3037] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.495012 28349 raft_consensus.cc:2454] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:30.476897 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05f786073084bd1b8c472c20dc8bff6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:30.495471 28353 raft_consensus.cc:2379] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d472d9f94d8949cf86284d32a161cebc in current term 2: Already voted for candidate d3a6b9b7913444d29c96581af6febf33 in this term.
I20240503 14:12:30.498275 28344 raft_consensus.cc:2454] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:30.459892 29828 ts_tablet_manager.cc:1888] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.501071 29828 log.cc:1196] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/c816f3b05cfb4496984f6cb28d310cc4
I20240503 14:12:30.501529 29828 ts_tablet_manager.cc:1909] T c816f3b05cfb4496984f6cb28d310cc4 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.484687 28600 ts_tablet_manager.cc:1398] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240503 14:12:30.492988 30272 raft_consensus.cc:482] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.502326 30272 raft_consensus.cc:504] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.502524 28600 tablet_bootstrap.cc:492] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.503074 30272 leader_election.cc:288] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.500368 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.504618 28600 tablet_bootstrap.cc:654] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.506736 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6696a237c3dd428097e2ebdf24afd63f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.509807 28600 tablet_bootstrap.cc:492] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.509975 28600 ts_tablet_manager.cc:1367] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:12:30.510912 28600 raft_consensus.cc:348] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.511174 28600 raft_consensus.cc:374] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.511236 28600 raft_consensus.cc:729] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.511049 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6696a237c3dd428097e2ebdf24afd63f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.511471 28213 raft_consensus.cc:2454] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.511492 28600 consensus_queue.cc:260] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.513492 28263 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:30.514026 28600 ts_tablet_manager.cc:1398] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:30.514422 28600 tablet_bootstrap.cc:492] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.515439 28212 raft_consensus.cc:2454] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 2.
I20240503 14:12:30.516180 28661 raft_consensus.cc:2229] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.516499 28661 raft_consensus.cc:2258] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.516620 28600 tablet_bootstrap.cc:654] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.517555 28661 ts_tablet_manager.cc:1875] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.521602 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 65e23e136ac94f9fa4180e15a81122a5
W20240503 14:12:30.522163 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 64318.2us
W20240503 14:12:30.522596 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d9c7b5fb9b4a483c8292f68387591034 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9c7b5fb9b4a483c8292f68387591034 already in progress: creating tablet
W20240503 14:12:30.523113 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet b476e87faea1415d95ae014a93fa22af because the tablet was not found. No further retry: Not found: Tablet not found: b476e87faea1415d95ae014a93fa22af
W20240503 14:12:30.523588 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet c816f3b05cfb4496984f6cb28d310cc4 because the tablet was not found. No further retry: Not found: Tablet not found: c816f3b05cfb4496984f6cb28d310cc4
W20240503 14:12:30.523958 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6de9e159d2ba47039dd6bd207a9cddeb because the tablet was not found. No further retry: Not found: Tablet not found: 6de9e159d2ba47039dd6bd207a9cddeb
W20240503 14:12:30.524420 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 007a892ef8504b848a400c05740eaa0c because the tablet was not found. No further retry: Not found: Tablet not found: 007a892ef8504b848a400c05740eaa0c
W20240503 14:12:30.524791 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 49445f0f1dd1451c9ba066e61273f2f1 because the tablet was not found. No further retry: Not found: Tablet not found: 49445f0f1dd1451c9ba066e61273f2f1
I20240503 14:12:30.525122 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet d7c79ffee3954b5c870f4a8aa6379517 (table default.Table_8f505fa519514278ae7489b9ff7bbaf2 [id=f58e9b32e96948cfab2760b64eb18b68]) successfully deleted
I20240503 14:12:30.525928 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 65e23e136ac94f9fa4180e15a81122a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.461158 28604 tablet_bootstrap.cc:492] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:30.492683 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d832cd30f2414303818d34a0cdf47e60" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.504042 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac752d8ab4fd4a218b052f388f416494" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:30.530046 28150 leader_election.cc:341] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.530233 30609 raft_consensus.cc:482] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.530277 28290 leader_election.cc:302] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:30.512521 28427 leader_election.cc:341] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.530536 30609 raft_consensus.cc:504] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.530911 28604 ts_tablet_manager.cc:1367] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.239s user 0.006s sys 0.000s
I20240503 14:12:30.531345 30609 leader_election.cc:288] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.531728 30609 raft_consensus.cc:2726] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.533304 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac752d8ab4fd4a218b052f388f416494" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:30.534726 28150 leader_election.cc:341] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.534869 28150 leader_election.cc:302] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.535159 30272 raft_consensus.cc:2726] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.541448 28604 raft_consensus.cc:348] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.541836 28604 raft_consensus.cc:374] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.549182 28614 ts_tablet_manager.cc:1398] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.109s user 0.002s sys 0.000s
I20240503 14:12:30.550191 30613 raft_consensus.cc:482] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.550359 30613 raft_consensus.cc:504] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.551460 30404 ts_tablet_manager.cc:1888] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.551606 30404 log.cc:1196] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a035a5249f8348b591b27ba3fa7ae43a
I20240503 14:12:30.552018 30404 ts_tablet_manager.cc:1909] T a035a5249f8348b591b27ba3fa7ae43a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:30.552204 28604 raft_consensus.cc:729] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.552527 28604 consensus_queue.cc:260] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.553134 28604 ts_tablet_manager.cc:1398] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 14:12:30.553447 28604 tablet_bootstrap.cc:492] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:30.554913 28604 tablet_bootstrap.cc:654] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.548570 28427 leader_election.cc:302] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:30.557582 30335 raft_consensus.cc:2781] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:30.557739 30335 raft_consensus.cc:686] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:30.557827 28427 leader_election.cc:302] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:30.558089 28427 leader_election.cc:341] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.558001 30335 consensus_queue.cc:237] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.559432 30451 raft_consensus.cc:2781] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:30.559557 30451 raft_consensus.cc:686] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:30.559835 30451 consensus_queue.cc:237] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.561671 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 49445f0f1dd1451c9ba066e61273f2f1 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
W20240503 14:12:30.562139 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet dbcc6cef7b7d464a9fbbae61caf0437d because the tablet was not found. No further retry: Not found: Tablet not found: dbcc6cef7b7d464a9fbbae61caf0437d
W20240503 14:12:30.562817 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a884cf4ea11e48ef941722b79ad50075 because the tablet was not found. No further retry: Not found: Tablet not found: a884cf4ea11e48ef941722b79ad50075
W20240503 14:12:30.563037 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet c9325d5142af4b018354bf38370b61e0 because the tablet was not found. No further retry: Not found: Tablet not found: c9325d5142af4b018354bf38370b61e0
W20240503 14:12:30.563239 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 3d54cc5c03034a6685c1da34f9bd65ac because the tablet was not found. No further retry: Not found: Tablet not found: 3d54cc5c03034a6685c1da34f9bd65ac
W20240503 14:12:30.563956 28427 leader_election.cc:341] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b476e87faea1415d95ae014a93fa22af
I20240503 14:12:30.564529 30446 consensus_queue.cc:1035] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.565404 30404 tablet_replica.cc:330] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:30.565652 30404 raft_consensus.cc:2229] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:30.565908 30404 pending_rounds.cc:62] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:30.565949 30404 pending_rounds.cc:68] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597770362564608 op_type: NO_OP noop_request { }
I20240503 14:12:30.566061 30404 raft_consensus.cc:2866] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:30.566133 30404 raft_consensus.cc:2258] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.566761 30404 ts_tablet_manager.cc:1875] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:30.568619 28427 consensus_peers.cc:485] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.568776 28427 leader_election.cc:302] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:30.569028 30566 raft_consensus.cc:2781] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:30.569087 30566 raft_consensus.cc:482] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:30.569139 30566 raft_consensus.cc:3037] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.571386 29828 tablet_replica.cc:330] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.571626 29828 raft_consensus.cc:2229] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.571822 29828 raft_consensus.cc:2258] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.571784 30566 raft_consensus.cc:504] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.572476 29828 ts_tablet_manager.cc:1875] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.572520 30566 leader_election.cc:288] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.572925 28427 leader_election.cc:302] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:30.573173 30566 raft_consensus.cc:2781] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:30.573271 30613 leader_election.cc:288] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.573369 30566 raft_consensus.cc:686] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [term 2 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:30.573424 30613 raft_consensus.cc:482] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.573539 30613 raft_consensus.cc:504] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.573644 30566 consensus_queue.cc:237] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:30.574795 27856 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 114074us
I20240503 14:12:30.576097 30613 leader_election.cc:288] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.577589 30610 tablet.cc:1722] T 69425d29d55b4fe4bb7f08ae390aaa86 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:30.578452 28614 tablet_bootstrap.cc:492] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:30.582239 28614 tablet_bootstrap.cc:654] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.582850 29828 ts_tablet_manager.cc:1888] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.584244 29828 log.cc:1196] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ce4846d662a64d51b16fd5447c371e9a
I20240503 14:12:30.584672 29828 ts_tablet_manager.cc:1909] T ce4846d662a64d51b16fd5447c371e9a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.586301 30446 consensus_queue.cc:1035] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.586876 29828 tablet_replica.cc:330] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.587494 29828 raft_consensus.cc:2229] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.588236 29828 raft_consensus.cc:2258] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:30.588954 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 58701.4us
I20240503 14:12:30.590193 29828 ts_tablet_manager.cc:1875] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.591801 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf810e84367f420b928a171790849445" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.592031 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 471992c1f7e84ef1953cd85448a492ec
I20240503 14:12:30.592732 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet aad397ed76ea4cf8820312ba1362512f from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:30.593521 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0c85614221c4e34811d02ecd2f7ce5d consensus state has no leader
I20240503 14:12:30.593847 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de9e159d2ba47039dd6bd207a9cddeb" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.593912 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f652e382ed2b40da8e5677659ce13416" } new_table_name: "default.Table_6667843423b24a5590e32de3c023d897" modify_external_catalogs: true
I20240503 14:12:30.596376 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9cad0a4cdfa8471bbd7066b91e85022e
I20240503 14:12:30.597218 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7d0ab9ea1144a03899dd81992e69db2" } new_table_name: "default.Table_3ebe7929a5eb414791bf6a704c6b10ef" modify_external_catalogs: true
I20240503 14:12:30.598747 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 34238c9859724d97a32f2a1d9315edb1
I20240503 14:12:30.599226 28032 catalog_manager.cc:6554] ReplaceTablet: tablet 34238c9859724d97a32f2a1d9315edb1 deleted and replaced by tablet 6e910e098ec44a07af6ed8ff4647fc7e
I20240503 14:12:30.599395 28032 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33344 (request call id 1247) took 1059 ms. Trace:
I20240503 14:12:30.599527 28032 rpcz_store.cc:276] 0503 14:12:29.539520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.026237 (+486717us) service_pool.cc:225] Handling call
0503 14:12:30.599313 (+573076us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:30.329992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:30.330079 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:30.330085 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:30.330088 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:30.330182 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:30.330192 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:30.330197 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:30.330261 (+ 64us) tablet.cc:699] Row locks acquired
0503 14:12:30.330264 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:30.330297 (+ 33us) write_op.cc:270] Start()
0503 14:12:30.330323 (+ 26us) write_op.cc:276] Timestamp: P: 1714745550330259 usec, L: 0
0503 14:12:30.330327 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:30.330469 (+ 142us) log.cc:842] Serialized 662 byte log entry
0503 14:12:30.590904 (+260435us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.592307 (+ 1403us) write_op.cc:301] APPLY: starting
0503 14:12:30.592321 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.592364 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.592387 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.592498 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.592552 (+ 54us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:30.592560 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:30.592625 (+ 65us) log.cc:842] Serialized 29 byte log entry
0503 14:12:30.592673 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.592690 (+ 17us) write_op.cc:454] Released schema lock
0503 14:12:30.592697 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.592753 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.592756 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1373,"apply.run_cpu_time_us":498,"apply.run_wall_time_us":497,"num_ops":2,"prepare.queue_time_us":180904,"prepare.run_cpu_time_us":525,"prepare.run_wall_time_us":524,"replication_time_us":260562}]]}
I20240503 14:12:30.600376 28025 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6fbf20a347f54724a0ce807436863785
I20240503 14:12:30.601126 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1e60a3851c8f4b49985e97e1d6096ac0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.601750 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e60a3851c8f4b49985e97e1d6096ac0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.607473 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a035a5249f8348b591b27ba3fa7ae43a (table default.Table_848d74c659274863a2c1046f6de2568e [id=ed5de25a209e4d3f944021a08a498d0e]) successfully deleted
I20240503 14:12:30.608255 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 471992c1f7e84ef1953cd85448a492ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.608968 30446 consensus_queue.cc:1035] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.609309 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c4e6f02aa70d498cabad3811f2fea24a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.609967 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4e6f02aa70d498cabad3811f2fea24a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.617508 30624 tablet.cc:1722] T 3e7b2865e9604ce58280321c7ee2a88e P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:30.619599 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 65e23e136ac94f9fa4180e15a81122a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.619755 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 471992c1f7e84ef1953cd85448a492ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.620230 28344 raft_consensus.cc:3037] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.620431 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cad0a4cdfa8471bbd7066b91e85022e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.621074 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 34238c9859724d97a32f2a1d9315edb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e910e098ec44a07af6ed8ff4647fc7e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.621722 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbf20a347f54724a0ce807436863785 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:30.622370 28290 leader_election.cc:341] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 007a892ef8504b848a400c05740eaa0c
W20240503 14:12:30.622649 28290 leader_election.cc:341] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: cf810e84367f420b928a171790849445
W20240503 14:12:30.623399 28290 leader_election.cc:341] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 007a892ef8504b848a400c05740eaa0c
I20240503 14:12:30.623507 28290 leader_election.cc:302] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.625249 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6696a237c3dd428097e2ebdf24afd63f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:30.625473 28207 raft_consensus.cc:3037] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.626241 28353 raft_consensus.cc:3037] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.626544 30315 raft_consensus.cc:2726] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.628963 28207 raft_consensus.cc:2454] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:30.637332 30623 tablet.cc:1722] T 69425d29d55b4fe4bb7f08ae390aaa86 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:30.627106 28290 leader_election.cc:341] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6de9e159d2ba47039dd6bd207a9cddeb
I20240503 14:12:30.640005 28600 tablet_bootstrap.cc:492] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.640244 28600 ts_tablet_manager.cc:1367] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.005s
I20240503 14:12:30.641412 28614 tablet_bootstrap.cc:492] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:30.641580 28614 ts_tablet_manager.cc:1367] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.004s
I20240503 14:12:30.641324 28600 raft_consensus.cc:348] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.641739 28600 raft_consensus.cc:374] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.641865 28600 raft_consensus.cc:729] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.642204 28600 consensus_queue.cc:260] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.642347 28614 raft_consensus.cc:348] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.642555 28614 raft_consensus.cc:374] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.642614 28614 raft_consensus.cc:729] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.642799 28614 consensus_queue.cc:260] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.643322 28600 ts_tablet_manager.cc:1398] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:30.639238 28290 leader_election.cc:302] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.643950 28661 ts_tablet_manager.cc:1888] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.644726 28600 tablet_bootstrap.cc:492] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:30.649106 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet c4b122081bd34391809a2c5746fad7e6 (table default.Table_8668db655be7429dbdc3ee1cd7512979 [id=4387686ed640413aa8ac41dcad75af40], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c4b122081bd34391809a2c5746fad7e6 consensus state has no leader
I20240503 14:12:30.651120 28600 tablet_bootstrap.cc:654] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.661285 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet c816f3b05cfb4496984f6cb28d310cc4 (table default.Table_94d6ea5674494c8a960eaee9c10ded44 [id=3c5661935acf4b3d8f3ccd4f4d7f84e2]) successfully deleted
I20240503 14:12:30.661659 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ce4846d662a64d51b16fd5447c371e9a (table default.Table_4fdbe31ba57a42b993d60fa19081c044 [id=ea4ef9402edd4db9820a6a318bfec37d]) successfully deleted
W20240503 14:12:30.661964 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet dbcc6cef7b7d464a9fbbae61caf0437d because the tablet was not found. No further retry: Not found: Tablet not found: dbcc6cef7b7d464a9fbbae61caf0437d
W20240503 14:12:30.662195 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e0175b38c1e0464a9a6208cfaf68ca0d because the tablet was not found. No further retry: Not found: Tablet not found: e0175b38c1e0464a9a6208cfaf68ca0d
W20240503 14:12:30.662441 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a884cf4ea11e48ef941722b79ad50075 because the tablet was not found. No further retry: Not found: Tablet not found: a884cf4ea11e48ef941722b79ad50075
W20240503 14:12:30.662652 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 20b176a071ba4f3fbff2a64d6e46027a because the tablet was not found. No further retry: Not found: Tablet not found: 20b176a071ba4f3fbff2a64d6e46027a
I20240503 14:12:30.663894 28661 log.cc:1196] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:30.652787 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 65e23e136ac94f9fa4180e15a81122a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.664407 28661 ts_tablet_manager.cc:1909] T 62e4f6e3908e420e9fa156ee24b437d6 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:30.665146 30626 tablet.cc:1722] T 69425d29d55b4fe4bb7f08ae390aaa86 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:30.668851 28540 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
W20240503 14:12:30.670591 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3366f813bd1f4df3843487ad40ba4203 consensus state has no leader
I20240503 14:12:30.671103 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_196b6e1c710b483889aef9e34ab1d8fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:30.671613 30623 tablet.cc:1722] T 3e7b2865e9604ce58280321c7ee2a88e P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240503 14:12:30.671746 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_196b6e1c710b483889aef9e34ab1d8fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:30.671994 28542 tablet.cc:2369] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:30.672432 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c99f41d735e4a92bda35ff68dca77c9 consensus state has no leader
I20240503 14:12:30.672894 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ddf2834af03d458b917b8fcbd0852de3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.673485 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddf2834af03d458b917b8fcbd0852de3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.674571 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1f1afeacc77b4bf4af1a39ce67f957b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.675186 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f1afeacc77b4bf4af1a39ce67f957b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.680258 28469 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbf20a347f54724a0ce807436863785 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.681129 28614 ts_tablet_manager.cc:1398] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20240503 14:12:30.681434 28614 tablet_bootstrap.cc:492] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:30.653337 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 34238c9859724d97a32f2a1d9315edb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e910e098ec44a07af6ed8ff4647fc7e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.653385 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 471992c1f7e84ef1953cd85448a492ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.653496 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cad0a4cdfa8471bbd7066b91e85022e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.681854 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1112) took 1093 ms. Trace:
I20240503 14:12:30.680675 30451 consensus_queue.cc:1035] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:12:30.656613 30335 raft_consensus.cc:482] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.682024 30335 raft_consensus.cc:504] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.682631 30335 leader_election.cc:288] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.682904 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet aad397ed76ea4cf8820312ba1362512f
I20240503 14:12:30.682862 28661 rpcz_store.cc:276] 0503 14:12:29.588184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.588246 (+ 62us) service_pool.cc:225] Handling call
0503 14:12:30.445893 (+857647us) ts_tablet_manager.cc:1120] Deleting tablet 62e4f6e3908e420e9fa156ee24b437d6
0503 14:12:30.445898 (+ 5us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.576013 (+130115us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:30.643818 (+ 67805us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:30.681833 (+ 38015us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":857518}
I20240503 14:12:30.683312 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1113) took 1094 ms. Trace:
I20240503 14:12:30.683422 28661 rpcz_store.cc:276] 0503 14:12:29.588577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.588712 (+ 135us) service_pool.cc:225] Handling call
0503 14:12:30.683179 (+1094467us) ts_tablet_manager.cc:1120] Deleting tablet 09f1f5216cc940888803ce8f6acd2a47
0503 14:12:30.683189 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.683296 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1094327}
I20240503 14:12:30.683509 28028 catalog_manager.cc:6554] ReplaceTablet: tablet aad397ed76ea4cf8820312ba1362512f deleted and replaced by tablet 8113a36f49ff4b29b32a0532d1aabebd
I20240503 14:12:30.683723 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1114) took 1082 ms. Trace:
I20240503 14:12:30.683779 28661 rpcz_store.cc:276] 0503 14:12:29.604527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.604942 (+ 415us) service_pool.cc:225] Handling call
0503 14:12:30.683625 (+1078683us) ts_tablet_manager.cc:1120] Deleting tablet 8cd037e135904e2cab09cbf39fed0b13
0503 14:12:30.683633 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.683710 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078250}
I20240503 14:12:30.683822 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet eff8442b6c02469eaedb08c51200829b from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:30.684031 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1115) took 1078 ms. Trace:
I20240503 14:12:30.684106 28661 rpcz_store.cc:276] 0503 14:12:29.605291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.605790 (+ 499us) service_pool.cc:225] Handling call
0503 14:12:30.683949 (+1078159us) ts_tablet_manager.cc:1120] Deleting tablet 0f642b306a8c49fd9c070a9d340889a2
0503 14:12:30.683956 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.684020 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077812}
I20240503 14:12:30.684384 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1116) took 1078 ms. Trace:
I20240503 14:12:30.684446 28661 rpcz_store.cc:276] 0503 14:12:29.606056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.606304 (+ 248us) service_pool.cc:225] Handling call
0503 14:12:30.684291 (+1077987us) ts_tablet_manager.cc:1120] Deleting tablet 310abf35ae02497c9d3696a0c49774ae
0503 14:12:30.684299 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.684370 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077838}
I20240503 14:12:30.684697 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1117) took 1078 ms. Trace:
I20240503 14:12:30.684748 28661 rpcz_store.cc:276] 0503 14:12:29.606700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.606736 (+ 36us) service_pool.cc:225] Handling call
0503 14:12:30.684618 (+1077882us) ts_tablet_manager.cc:1120] Deleting tablet 9a1bc119df7d4694bbe926611ef73245
0503 14:12:30.684625 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.684686 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077748}
I20240503 14:12:30.684978 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1118) took 1077 ms. Trace:
I20240503 14:12:30.686942 28614 tablet_bootstrap.cc:654] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.680514 28604 tablet_bootstrap.cc:492] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:30.689394 28604 ts_tablet_manager.cc:1367] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.136s user 0.001s sys 0.009s
I20240503 14:12:30.689708 29828 ts_tablet_manager.cc:1888] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.689823 29828 log.cc:1196] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/920b41e3b21347b393497de4d4e87469
I20240503 14:12:30.690117 29828 ts_tablet_manager.cc:1909] T 920b41e3b21347b393497de4d4e87469 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.690996 29828 tablet_replica.cc:330] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.691232 29828 raft_consensus.cc:2229] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:30.691529 29828 pending_rounds.cc:62] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:30.691576 29828 pending_rounds.cc:68] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597770332913664 op_type: NO_OP noop_request { }
I20240503 14:12:30.691690 29828 raft_consensus.cc:2866] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:30.691774 29828 raft_consensus.cc:2258] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.693177 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_38b33f45a641422ba567e4bbadaa6e23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:30.693543 30404 ts_tablet_manager.cc:1888] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.693662 30404 log.cc:1196] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/646f83185d5241268d4126d10561e937
W20240503 14:12:30.693919 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38b33f45a641422ba567e4bbadaa6e23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.694015 30404 ts_tablet_manager.cc:1909] T 646f83185d5241268d4126d10561e937 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:30.694504 30404 tablet_replica.cc:330] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:30.694733 30404 raft_consensus.cc:2229] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.694908 30404 raft_consensus.cc:2258] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.695473 30404 ts_tablet_manager.cc:1875] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.690189 28604 raft_consensus.cc:348] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.700273 28604 raft_consensus.cc:374] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.700348 28604 raft_consensus.cc:729] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.691120 28661 rpcz_store.cc:276] 0503 14:12:29.607014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.607050 (+ 36us) service_pool.cc:225] Handling call
0503 14:12:30.684904 (+1077854us) ts_tablet_manager.cc:1120] Deleting tablet dc31b2e907b04ea19a6f9f7fd8f57843
0503 14:12:30.684911 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.684967 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077734}
I20240503 14:12:30.701143 30335 raft_consensus.cc:482] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.701308 30335 raft_consensus.cc:504] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.702070 30335 leader_election.cc:288] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.702214 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1119) took 1093 ms. Trace:
I20240503 14:12:30.702325 28661 rpcz_store.cc:276] 0503 14:12:29.607385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.607426 (+ 41us) service_pool.cc:225] Handling call
0503 14:12:30.700836 (+1093410us) ts_tablet_manager.cc:1120] Deleting tablet eff8442b6c02469eaedb08c51200829b
0503 14:12:30.700850 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:30.700984 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1093269}
I20240503 14:12:30.702924 28661 tablet_replica.cc:330] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:30.703177 28661 raft_consensus.cc:2229] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:30.703435 28661 pending_rounds.cc:62] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:30.703482 28661 pending_rounds.cc:68] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597775095566336 op_type: NO_OP noop_request { }
I20240503 14:12:30.703572 28661 raft_consensus.cc:2866] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:30.703644 28661 raft_consensus.cc:2258] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.704783 28661 ts_tablet_manager.cc:1875] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.705278 30272 raft_consensus.cc:482] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.705494 30272 raft_consensus.cc:504] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.706137 30272 leader_election.cc:288] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.706888 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26e8389639748ae99832d7a7a9d5af3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.707048 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26e8389639748ae99832d7a7a9d5af3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:30.708323 28150 leader_election.cc:341] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:30.708559 28150 leader_election.cc:341] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.708647 28150 leader_election.cc:302] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.709043 30272 raft_consensus.cc:2726] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:30.680735 28427 consensus_peers.cc:485] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6de9e159d2ba47039dd6bd207a9cddeb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.711055 28427 leader_election.cc:302] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:30.711337 30335 raft_consensus.cc:2781] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:30.711470 30335 raft_consensus.cc:686] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:30.711748 30335 consensus_queue.cc:237] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.712950 30616 raft_consensus.cc:482] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.713124 30616 raft_consensus.cc:504] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.714088 30616 leader_election.cc:288] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.716662 29828 ts_tablet_manager.cc:1875] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.716936 28604 consensus_queue.cc:260] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.718364 28604 ts_tablet_manager.cc:1398] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240503 14:12:30.718698 28604 tablet_bootstrap.cc:492] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:30.719568 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 92459.5us
I20240503 14:12:30.719880 28290 leader_election.cc:302] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.720199 30315 raft_consensus.cc:2726] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.720402 30315 raft_consensus.cc:2726] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.720798 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cad0a4cdfa8471bbd7066b91e85022e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.720927 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 34238c9859724d97a32f2a1d9315edb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e910e098ec44a07af6ed8ff4647fc7e) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.721398 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbf20a347f54724a0ce807436863785 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.722862 28600 tablet_bootstrap.cc:492] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.723026 28600 ts_tablet_manager.cc:1367] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.078s user 0.021s sys 0.009s
I20240503 14:12:30.723958 28600 raft_consensus.cc:348] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.724226 28600 raft_consensus.cc:374] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.725271 28600 raft_consensus.cc:729] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.725522 28600 consensus_queue.cc:260] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.726246 28600 ts_tablet_manager.cc:1398] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:30.726569 28600 tablet_bootstrap.cc:492] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.728602 28600 tablet_bootstrap.cc:654] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.735472 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a035a5249f8348b591b27ba3fa7ae43a" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.736941 28353 raft_consensus.cc:3037] T 33ee9b69dd1f47219e7d3126207810c7 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:30.737952 28290 leader_election.cc:341] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a035a5249f8348b591b27ba3fa7ae43a
I20240503 14:12:30.738157 28290 leader_election.cc:302] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.738430 29828 ts_tablet_manager.cc:1888] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.738529 30616 raft_consensus.cc:2726] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:30.740021 30404 ts_tablet_manager.cc:1888] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.742148 30404 log.cc:1196] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/6696a237c3dd428097e2ebdf24afd63f
I20240503 14:12:30.742488 30404 ts_tablet_manager.cc:1909] T 6696a237c3dd428097e2ebdf24afd63f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:30.742565 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6696a237c3dd428097e2ebdf24afd63f" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:30.743571 30626 tablet.cc:1722] T 86d810ab0cf34941a343eef1593632bf P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:30.745349 30404 tablet_replica.cc:330] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:30.745555 30404 raft_consensus.cc:2229] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.745728 30404 raft_consensus.cc:2258] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.748133 30272 raft_consensus.cc:482] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.748303 30272 raft_consensus.cc:504] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.738529 29828 log.cc:1196] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/13c7a51c68ef478e9bdc5b3e3dc3d5df
I20240503 14:12:30.749023 30272 leader_election.cc:288] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:30.749140 30272 raft_consensus.cc:482] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.749289 29828 ts_tablet_manager.cc:1909] T 13c7a51c68ef478e9bdc5b3e3dc3d5df P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:30.749248 30272 raft_consensus.cc:504] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.749747 30272 leader_election.cc:288] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.751294 30404 ts_tablet_manager.cc:1875] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.753158 28600 tablet_bootstrap.cc:492] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:30.753319 28600 ts_tablet_manager.cc:1367] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240503 14:12:30.753547 30566 consensus_queue.cc:1035] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.754348 28600 raft_consensus.cc:348] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.754577 28600 raft_consensus.cc:374] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.754631 28600 raft_consensus.cc:729] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.754835 28600 consensus_queue.cc:260] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.755717 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdc868ddf0a490292eae78ebc8c8ec6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.755904 28207 raft_consensus.cc:2454] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.757436 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:30.757685 28215 raft_consensus.cc:2454] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.758119 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdc868ddf0a490292eae78ebc8c8ec6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.766360 28600 ts_tablet_manager.cc:1398] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:12:30.766794 28600 tablet_bootstrap.cc:492] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:30.769192 28600 tablet_bootstrap.cc:654] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.771358 28604 tablet_bootstrap.cc:654] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.769918 30632 tablet.cc:1722] T 86d810ab0cf34941a343eef1593632bf P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:30.776964 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f75e102dd4249c3800e1fcb4339acf3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:30.777724 28150 leader_election.cc:341] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.777678 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f75e102dd4249c3800e1fcb4339acf3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.777856 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cc84cf703048148abfe03b7dee12ce" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:30.778184 28150 leader_election.cc:341] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.778292 28150 leader_election.cc:302] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:30.778860 28150 leader_election.cc:341] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.779492 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet aad397ed76ea4cf8820312ba1362512f with delete_type TABLET_DATA_DELETED (replaced by tablet 8113a36f49ff4b29b32a0532d1aabebd) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:30.779538 30638 raft_consensus.cc:2726] T 0f75e102dd4249c3800e1fcb4339acf3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.780100 29828 tablet_replica.cc:330] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:30.780776 29828 raft_consensus.cc:2229] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.782373 30446 consensus_queue.cc:1035] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.780267 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.779974 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cc84cf703048148abfe03b7dee12ce" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.780146 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet aad397ed76ea4cf8820312ba1362512f with delete_type TABLET_DATA_DELETED (replaced by tablet 8113a36f49ff4b29b32a0532d1aabebd) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.783293 30446 consensus_queue.cc:1035] T b59c0bc115d949f69d80e1f3628d5a79 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:30.783509 29828 raft_consensus.cc:2258] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.784191 29828 ts_tablet_manager.cc:1875] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.785015 30446 consensus_queue.cc:1035] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:30.785759 28427 leader_election.cc:341] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:30.786798 28150 leader_election.cc:341] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.786945 28150 leader_election.cc:302] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:30.788041 30640 raft_consensus.cc:2726] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:30.791335 30404 ts_tablet_manager.cc:1888] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:30.793124 30446 consensus_queue.cc:1035] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:30.793843 28427 leader_election.cc:341] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:30.794646 28427 leader_election.cc:341] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.796154 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet aad397ed76ea4cf8820312ba1362512f with delete_type TABLET_DATA_DELETED (replaced by tablet 8113a36f49ff4b29b32a0532d1aabebd) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:30.802387 30610 tablet.cc:1722] T 3e7b2865e9604ce58280321c7ee2a88e P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240503 14:12:30.802814 28661 ts_tablet_manager.cc:1888] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.802949 28661 log.cc:1196] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/b476e87faea1415d95ae014a93fa22af
I20240503 14:12:30.803333 28661 ts_tablet_manager.cc:1909] T b476e87faea1415d95ae014a93fa22af P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:30.804344 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 920b41e3b21347b393497de4d4e87469 (table default.Table_28c16c4a952d45c0b34fd1d4864ac748 [id=2fd597e69c1041a2984ca535058f6fab]) successfully deleted
I20240503 14:12:30.804638 28661 tablet_replica.cc:330] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:30.804684 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 13c7a51c68ef478e9bdc5b3e3dc3d5df (table default.Table_9622df66720f484b8f46a671ad60bf5d [id=62af87809548445e93596d9a6030f826]) successfully deleted
I20240503 14:12:30.804847 28661 raft_consensus.cc:2229] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:30.805013 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet d9c7b5fb9b4a483c8292f68387591034 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9c7b5fb9b4a483c8292f68387591034 already in progress: creating tablet
I20240503 14:12:30.805137 28661 raft_consensus.cc:2258] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:30.805246 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 646f83185d5241268d4126d10561e937 (table default.Table_4d256c29f9b7400497caee67285a9831 [id=6c9175a58b8544798ce769460dfd9cf5]) successfully deleted
I20240503 14:12:30.805414 28427 leader_election.cc:302] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:30.805488 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 6696a237c3dd428097e2ebdf24afd63f (table default.Table_43c9760cf3b54d68b218cafe8393cda2 [id=821a0387d6f44024b05959350fd53b63]) successfully deleted
I20240503 14:12:30.805686 30446 raft_consensus.cc:2781] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:30.805757 30446 raft_consensus.cc:482] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:30.805776 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 62e4f6e3908e420e9fa156ee24b437d6 (table default.Table_5654996201694c1b90fc6e728e068ca3 [id=9a6689ece91646069be1867e8f141392]) successfully deleted
I20240503 14:12:30.805821 30446 raft_consensus.cc:3037] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:30.806044 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 09f1f5216cc940888803ce8f6acd2a47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09f1f5216cc940888803ce8f6acd2a47 already in progress: creating tablet
I20240503 14:12:30.808691 30446 raft_consensus.cc:504] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.809423 30446 leader_election.cc:288] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.810249 28427 leader_election.cc:302] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:30.806285 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 8cd037e135904e2cab09cbf39fed0b13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cd037e135904e2cab09cbf39fed0b13 already in progress: creating tablet
W20240503 14:12:30.813227 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 0f642b306a8c49fd9c070a9d340889a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f642b306a8c49fd9c070a9d340889a2 already in progress: creating tablet
W20240503 14:12:30.814638 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 310abf35ae02497c9d3696a0c49774ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 310abf35ae02497c9d3696a0c49774ae already in progress: creating tablet
I20240503 14:12:30.814893 30446 raft_consensus.cc:2781] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:30.814983 30446 raft_consensus.cc:482] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:30.815058 30446 raft_consensus.cc:3037] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:30.815886 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9a1bc119df7d4694bbe926611ef73245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a1bc119df7d4694bbe926611ef73245 already in progress: creating tablet
I20240503 14:12:30.816658 30610 tablet.cc:1722] T 86d810ab0cf34941a343eef1593632bf P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 14:12:30.816941 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet dc31b2e907b04ea19a6f9f7fd8f57843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc31b2e907b04ea19a6f9f7fd8f57843 already in progress: creating tablet
I20240503 14:12:30.818042 30446 raft_consensus.cc:504] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:30.817971 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet eff8442b6c02469eaedb08c51200829b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff8442b6c02469eaedb08c51200829b already in progress: creating tablet
I20240503 14:12:30.818778 30446 leader_election.cc:288] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:30.819058 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet d9c7b5fb9b4a483c8292f68387591034 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9c7b5fb9b4a483c8292f68387591034 already in progress: creating tablet
I20240503 14:12:30.821525 28661 ts_tablet_manager.cc:1875] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:30.822376 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d933cabbaf9146438f67771682480e59" } new_table_name: "default.Table_ef5776d7c73644dc86566abb31f89bb8" modify_external_catalogs: true
I20240503 14:12:30.822767 30626 tablet.cc:1722] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:30.831670 28345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:30.831889 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet b476e87faea1415d95ae014a93fa22af (table default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]) successfully deleted
I20240503 14:12:30.832712 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdc868ddf0a490292eae78ebc8c8ec6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:30.833925 28427 leader_election.cc:341] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:30.834146 28427 leader_election.cc:341] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.840903 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2c0811e5c5b3481bbd30068c14bde1b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.841643 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c0811e5c5b3481bbd30068c14bde1b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.843667 30626 tablet.cc:1722] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:30.843734 28025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_d67437cde20547fb8188102d8a8e9d54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.844362 28025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67437cde20547fb8188102d8a8e9d54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.846069 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5ef9d168912f483c8004a5bbfe02e07d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.846771 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ef9d168912f483c8004a5bbfe02e07d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.847591 30335 raft_consensus.cc:482] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.847743 30335 raft_consensus.cc:504] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.848428 30335 leader_election.cc:288] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:30.848533 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_390a8ae260bb40809e52f71153fa46f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:30.849243 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_390a8ae260bb40809e52f71153fa46f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:30.851800 28604 tablet_bootstrap.cc:492] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:30.851954 28604 ts_tablet_manager.cc:1367] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.133s user 0.002s sys 0.002s
I20240503 14:12:30.852905 28604 raft_consensus.cc:348] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.853119 28604 raft_consensus.cc:374] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.853165 28604 raft_consensus.cc:729] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.853459 28604 consensus_queue.cc:260] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.854483 28604 ts_tablet_manager.cc:1398] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:30.854811 28604 tablet_bootstrap.cc:492] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:30.856777 28604 tablet_bootstrap.cc:654] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.866952 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2]). Expected version 3 got 0
I20240503 14:12:30.867624 30608 raft_consensus.cc:482] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.867820 30608 raft_consensus.cc:504] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:30.868777 30608 leader_election.cc:288] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.869309 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:30.867462 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:30.870420 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 86d810ab0cf34941a343eef1593632bf (table default.Table_9f2d3cc9c6724dab9ba35ff24e85c236 [id=c358e28088d44cf598c96a1bd2a4321e]). Expected version 1 got 0
I20240503 14:12:30.870873 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 888636e9beca43edab96a8736751409e (table default.Table_a13cfbd07ce348578e9883c1205e8b9b [id=3854b08f7cd348c897aac4aacb7f5d06]). Expected version 3 got 0
I20240503 14:12:30.871341 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 69425d29d55b4fe4bb7f08ae390aaa86 (table default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e]). Expected version 2 got 0
I20240503 14:12:30.871757 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_392e46b937cf45e58d01188a21cf276c [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 1 got 0
I20240503 14:12:30.872121 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 2f7b805aee554f8fa4c198ce3409c6e6 (table default.Table_ee9f2d2bd2b54111963a23deaf110216 [id=4df57b62a5b34ac182e799495a848041]). Expected version 1 got 0
I20240503 14:12:30.872746 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 5ac7b6df20b14d1ead036cdc42174d3a (table default.Table_bdce2849aeb843c0a01b36cd7dbcaee6 [id=0edef09212734f47be4ac77cca4fc186]). Expected version 2 got 0
I20240503 14:12:30.873492 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:30.875636 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (table default.Table_bfff85704d88463383cbf94f9b121359 [id=bd1950deb4274d86bf288d501cf5c710]). Expected version 2 got 0
I20240503 14:12:30.876402 28024 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
W20240503 14:12:30.877027 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f223cbbcaa5402a888b084df0fe1c27 consensus state has no leader
I20240503 14:12:30.877259 28030 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1243) took 1897 ms. Trace:
I20240503 14:12:30.875303 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac752d8ab4fd4a218b052f388f416494" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:30.877574 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 74d23697414b4a3eb0c6a0caf86bf64d (DEFAULT_TABLE table=default.Table_5676e58199804bb9ad655f7fe44d0a9e [id=9582c9479e0747368ffa410c03d8856b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.877377 28030 rpcz_store.cc:276] 0503 14:12:28.979252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.820888 (+841636us) service_pool.cc:225] Handling call
0503 14:12:29.821311 (+ 423us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:29.821524 (+ 213us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:29.821687 (+ 163us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:29.821750 (+ 63us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:30.024184 (+202434us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:30.877170 (+852986us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:29.824543 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:29.824695 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:29.824702 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:29.824705 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:29.824820 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:29.824833 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:29.824838 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:29.824920 (+ 82us) tablet.cc:699] Row locks acquired
0503 14:12:29.824922 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:29.824963 (+ 41us) write_op.cc:270] Start()
0503 14:12:29.825000 (+ 37us) write_op.cc:276] Timestamp: P: 1714745549824927 usec, L: 0
0503 14:12:29.825003 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:29.825175 (+ 172us) log.cc:842] Serialized 405 byte log entry
0503 14:12:30.021209 (+196034us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.022522 (+ 1313us) write_op.cc:301] APPLY: starting
0503 14:12:30.022533 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.022582 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.022604 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.022650 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.022678 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:30.022685 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:30.022755 (+ 70us) log.cc:842] Serialized 23 byte log entry
0503 14:12:30.022805 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.022819 (+ 14us) write_op.cc:454] Released schema lock
0503 14:12:30.022824 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.022877 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.022881 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1285,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":401,"num_ops":1,"prepare.queue_time_us":2204,"prepare.run_cpu_time_us":696,"prepare.run_wall_time_us":697,"replication_time_us":196194}]]}
I20240503 14:12:30.878110 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d23697414b4a3eb0c6a0caf86bf64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.878214 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 5757e6e8703640519f7d38754cc0c8ce from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:30.878844 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet 61da48859c4d4a73bb1b9d7c0ede8106 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:30.881917 28427 leader_election.cc:341] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:30.885462 28661 ts_tablet_manager.cc:1888] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:30.886171 28033 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 3e7b2865e9604ce58280321c7ee2a88e (table default.Table_ca6cee392d934dd3ac773e100ae031d2 [id=32be054076954aed9265888ca62e510b]). Expected version 4 got 0
I20240503 14:12:30.890929 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_aa95e49d837a43cf9ed5ce0f5858cc45" } modify_external_catalogs: true
I20240503 14:12:30.891242 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_aa95e49d837a43cf9ed5ce0f5858cc45" } modify_external_catalogs: true
I20240503 14:12:30.891506 28661 log.cc:1196] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a31555fc800142bd849dbd572a194002
I20240503 14:12:30.891906 28661 ts_tablet_manager.cc:1909] T a31555fc800142bd849dbd572a194002 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:30.895573 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 3e7b2865e9604ce58280321c7ee2a88e (table default.Table_ca6cee392d934dd3ac773e100ae031d2 [id=32be054076954aed9265888ca62e510b]). Expected version 4 got 0
I20240503 14:12:30.895942 28026 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet d832cd30f2414303818d34a0cdf47e60 (table default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2]). Expected version 3 got 0
I20240503 14:12:30.896296 28026 catalog_manager.cc:5484] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:30.897075 28026 catalog_manager.cc:5484] T 33ee9b69dd1f47219e7d3126207810c7 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:30.916749 28614 tablet_bootstrap.cc:492] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:30.917191 28614 ts_tablet_manager.cc:1367] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.236s user 0.008s sys 0.011s
I20240503 14:12:30.924065 28604 tablet_bootstrap.cc:492] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:30.926743 28604 ts_tablet_manager.cc:1367] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.002s
I20240503 14:12:30.928694 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 2551ba0c77104486897873a9d78933a8 (DEFAULT_TABLE table=default.Table_0561c91abcaf4d26b3f20f19bbc4af18 [id=62579bbf7bd2414dba0b108c35e55f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.929168 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea649e25074468ea487f77c1a96d79e (DEFAULT_TABLE table=default.Table_076b0d07ce0c4cba9fbf355693ab1aa9 [id=a3b81b0c395b4095b822824056a4e8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.929553 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 66936337854b4f6f921a6c66d2cf2a95 (DEFAULT_TABLE table=default.Table_d501859ed20b479c84d1abf619d54d9f [id=e6ec6b5c15e54558b36c611b7b53abb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.930002 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 7de83e2102924b8884fbf20300058f43 (DEFAULT_TABLE table=default.Table_2841ebe2e868418690799c3ccd7901df [id=b292d157f70c41d7a3220b88a7cd83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.930094 28330 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.929980 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 4a28d2b3e5084a1b93f1074d100e397a (DEFAULT_TABLE table=default.Table_b3822edda94b4fd089b7e41145d1f6af [id=c631de85e8d14c29b2ec6a13ac00d56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.933876 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.934202 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.934427 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:30.938274 28320 tablet_service.cc:1467] Processing CreateTablet for tablet a9b7aee2f7fe4a37ac7b97250a9e7fe6 (DEFAULT_TABLE table=default.Table_b22b2e286534472a9988dfb13d282282 [id=bb5260d046e441678a7f089f1da3ea5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.938869 28333 tablet_service.cc:1467] Processing CreateTablet for tablet c8087dc9696f46dc9dc8c23a947c7bfe (DEFAULT_TABLE table=default.Table_00f5a3c1cea944eb91d3b4655518ae6b [id=f304a1801c6446b59ef5074490d489eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.939900 28604 raft_consensus.cc:348] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.941152 28604 raft_consensus.cc:374] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.941620 28604 raft_consensus.cc:729] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.942595 28604 consensus_queue.cc:260] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:30.945981 28427 consensus_peers.cc:485] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.951527 28604 ts_tablet_manager.cc:1398] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20240503 14:12:30.956965 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2551ba0c77104486897873a9d78933a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.962581 28356 raft_consensus.cc:3037] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:30.967396 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 74d23697414b4a3eb0c6a0caf86bf64d (DEFAULT_TABLE table=default.Table_5676e58199804bb9ad655f7fe44d0a9e [id=9582c9479e0747368ffa410c03d8856b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.967890 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d23697414b4a3eb0c6a0caf86bf64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.968015 28462 tablet_service.cc:1467] Processing CreateTablet for tablet c8087dc9696f46dc9dc8c23a947c7bfe (DEFAULT_TABLE table=default.Table_00f5a3c1cea944eb91d3b4655518ae6b [id=f304a1801c6446b59ef5074490d489eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.968544 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8087dc9696f46dc9dc8c23a947c7bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.972546 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 2551ba0c77104486897873a9d78933a8 (DEFAULT_TABLE table=default.Table_0561c91abcaf4d26b3f20f19bbc4af18 [id=62579bbf7bd2414dba0b108c35e55f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.973057 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2551ba0c77104486897873a9d78933a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.976486 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea649e25074468ea487f77c1a96d79e (DEFAULT_TABLE table=default.Table_076b0d07ce0c4cba9fbf355693ab1aa9 [id=a3b81b0c395b4095b822824056a4e8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.976984 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea649e25074468ea487f77c1a96d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.978530 28614 raft_consensus.cc:348] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.979532 28614 raft_consensus.cc:374] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:30.979779 28614 raft_consensus.cc:729] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.980459 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 66936337854b4f6f921a6c66d2cf2a95 (DEFAULT_TABLE table=default.Table_d501859ed20b479c84d1abf619d54d9f [id=e6ec6b5c15e54558b36c611b7b53abb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.980473 28614 consensus_queue.cc:260] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:30.980901 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66936337854b4f6f921a6c66d2cf2a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.981925 28614 ts_tablet_manager.cc:1398] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.064s user 0.003s sys 0.001s
I20240503 14:12:30.984463 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 7de83e2102924b8884fbf20300058f43 (DEFAULT_TABLE table=default.Table_2841ebe2e868418690799c3ccd7901df [id=b292d157f70c41d7a3220b88a7cd83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.984970 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de83e2102924b8884fbf20300058f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.986999 28604 tablet_bootstrap.cc:492] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:30.988462 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 4a28d2b3e5084a1b93f1074d100e397a (DEFAULT_TABLE table=default.Table_b3822edda94b4fd089b7e41145d1f6af [id=c631de85e8d14c29b2ec6a13ac00d56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.988925 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a28d2b3e5084a1b93f1074d100e397a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.992465 28465 tablet_service.cc:1467] Processing CreateTablet for tablet a9b7aee2f7fe4a37ac7b97250a9e7fe6 (DEFAULT_TABLE table=default.Table_b22b2e286534472a9988dfb13d282282 [id=bb5260d046e441678a7f089f1da3ea5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.992924 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b7aee2f7fe4a37ac7b97250a9e7fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.003650 28661 tablet_replica.cc:330] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:31.003963 28661 raft_consensus.cc:2229] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:30.792672 30640 raft_consensus.cc:482] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.792959 28600 tablet_bootstrap.cc:492] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.004401 28600 ts_tablet_manager.cc:1367] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.238s user 0.004s sys 0.000s
I20240503 14:12:30.830852 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdc868ddf0a490292eae78ebc8c8ec6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:30.831524 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:31.004565 28216 raft_consensus.cc:3037] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.004583 28209 raft_consensus.cc:3037] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.005465 28600 raft_consensus.cc:348] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.005712 28600 raft_consensus.cc:374] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.005762 28600 raft_consensus.cc:729] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.005951 28600 consensus_queue.cc:260] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.006520 28600 ts_tablet_manager.cc:1398] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:31.006860 28600 tablet_bootstrap.cc:492] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.007331 28216 raft_consensus.cc:2454] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:31.007440 28209 raft_consensus.cc:2454] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:30.833355 30638 raft_consensus.cc:482] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.007721 30638 raft_consensus.cc:504] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.004282 30404 log.cc:1196] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/31ddd791d4e14630bfb3b4b1513f76e1
I20240503 14:12:31.008571 30638 leader_election.cc:288] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:31.008642 30404 ts_tablet_manager.cc:1909] T 31ddd791d4e14630bfb3b4b1513f76e1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:30.835813 30623 tablet.cc:1722] T 8acbc154c03c47c1b2f9f433fcb2ef9a P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:31.009137 30404 tablet_replica.cc:330] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.009317 28600 tablet_bootstrap.cc:654] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.009347 30404 raft_consensus.cc:2229] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:30.857846 30272 raft_consensus.cc:482] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:30.869240 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ddd791d4e14630bfb3b4b1513f76e1" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:31.009547 30272 raft_consensus.cc:504] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.010224 30272 leader_election.cc:288] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.859457 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac752d8ab4fd4a218b052f388f416494" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:31.010416 28208 raft_consensus.cc:2454] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:30.912302 30641 raft_consensus.cc:482] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.012461 30641 raft_consensus.cc:504] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.012938 28661 raft_consensus.cc:2258] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.013217 30641 leader_election.cc:288] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:30.928112 28187 tablet_service.cc:1467] Processing CreateTablet for tablet 7de83e2102924b8884fbf20300058f43 (DEFAULT_TABLE table=default.Table_2841ebe2e868418690799c3ccd7901df [id=b292d157f70c41d7a3220b88a7cd83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.013593 28661 ts_tablet_manager.cc:1875] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.013657 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de83e2102924b8884fbf20300058f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.928184 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 74d23697414b4a3eb0c6a0caf86bf64d (DEFAULT_TABLE table=default.Table_5676e58199804bb9ad655f7fe44d0a9e [id=9582c9479e0747368ffa410c03d8856b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.016808 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d23697414b4a3eb0c6a0caf86bf64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.020499 28604 tablet_bootstrap.cc:654] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:30.928550 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 2551ba0c77104486897873a9d78933a8 (DEFAULT_TABLE table=default.Table_0561c91abcaf4d26b3f20f19bbc4af18 [id=62579bbf7bd2414dba0b108c35e55f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.022110 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2551ba0c77104486897873a9d78933a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.928946 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea649e25074468ea487f77c1a96d79e (DEFAULT_TABLE table=default.Table_076b0d07ce0c4cba9fbf355693ab1aa9 [id=a3b81b0c395b4095b822824056a4e8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.028980 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea649e25074468ea487f77c1a96d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.922811 30632 tablet.cc:1722] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:30.929670 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 4a28d2b3e5084a1b93f1074d100e397a (DEFAULT_TABLE table=default.Table_b3822edda94b4fd089b7e41145d1f6af [id=c631de85e8d14c29b2ec6a13ac00d56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.030740 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a28d2b3e5084a1b93f1074d100e397a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.929302 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 66936337854b4f6f921a6c66d2cf2a95 (DEFAULT_TABLE table=default.Table_d501859ed20b479c84d1abf619d54d9f [id=e6ec6b5c15e54558b36c611b7b53abb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.936726 28185 tablet_service.cc:1467] Processing CreateTablet for tablet a9b7aee2f7fe4a37ac7b97250a9e7fe6 (DEFAULT_TABLE table=default.Table_b22b2e286534472a9988dfb13d282282 [id=bb5260d046e441678a7f089f1da3ea5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:30.984999 30655 raft_consensus.cc:482] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.004271 30640 raft_consensus.cc:504] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.032917 30655 raft_consensus.cc:504] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.033700 30655 leader_election.cc:288] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:31.033735 30640 leader_election.cc:288] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:30.936056 28179 tablet_service.cc:1467] Processing CreateTablet for tablet c8087dc9696f46dc9dc8c23a947c7bfe (DEFAULT_TABLE table=default.Table_00f5a3c1cea944eb91d3b4655518ae6b [id=f304a1801c6446b59ef5074490d489eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.037133 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66936337854b4f6f921a6c66d2cf2a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.037742 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8087dc9696f46dc9dc8c23a947c7bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.038036 28487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b94092725fa4956a4c00e3685f9aeaf" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.038214 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b52ef7ecedf4273939406581d843980" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.038480 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b7aee2f7fe4a37ac7b97250a9e7fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.038982 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea261d8d5d174bee804aa2e171f35008" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.039310 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a9d58fb30d4b60ae0a44ec217fea11" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:31.039817 28150 leader_election.cc:341] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.040048 28150 leader_election.cc:341] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.040366 28150 leader_election.cc:341] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.040560 28150 leader_election.cc:341] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.038249 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fe462c9ee54a27b9714f6457036d76" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.041901 29828 ts_tablet_manager.cc:1888] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:31.043124 28150 leader_election.cc:341] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.043671 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b52ef7ecedf4273939406581d843980" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.044180 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a9d58fb30d4b60ae0a44ec217fea11" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.044436 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fe462c9ee54a27b9714f6457036d76" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.043412 29828 log.cc:1196] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/9ac2833a0bb045dcb2824468d936ffe4
I20240503 14:12:31.044834 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea261d8d5d174bee804aa2e171f35008" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.044303 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b94092725fa4956a4c00e3685f9aeaf" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.045615 30640 raft_consensus.cc:482] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.045778 30640 raft_consensus.cc:504] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.046479 30640 leader_election.cc:288] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:31.048507 29828 ts_tablet_manager.cc:1909] T 9ac2833a0bb045dcb2824468d936ffe4 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.053263 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00dbc5a90674873b61b377e3e8d92fb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:31.053282 28150 leader_election.cc:341] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.053740 28150 leader_election.cc:302] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.060709 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00dbc5a90674873b61b377e3e8d92fb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.062072 30646 tablet.cc:1722] T 8acbc154c03c47c1b2f9f433fcb2ef9a P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:31.075711 30566 raft_consensus.cc:482] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.075888 30566 raft_consensus.cc:504] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.076431 28604 tablet_bootstrap.cc:492] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.076596 30566 leader_election.cc:288] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.076802 28604 ts_tablet_manager.cc:1367] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.000s
I20240503 14:12:31.079713 28661 ts_tablet_manager.cc:1888] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.080286 28614 tablet_bootstrap.cc:492] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.086877 30640 raft_consensus.cc:2726] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:31.087671 28150 leader_election.cc:341] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.088143 28150 leader_election.cc:302] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.088847 30641 raft_consensus.cc:2726] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.089836 29828 tablet_replica.cc:330] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
W20240503 14:12:31.090548 28150 leader_election.cc:341] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.090752 29828 raft_consensus.cc:2229] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.091375 28150 leader_election.cc:302] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.092329 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de83e2102924b8884fbf20300058f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.092974 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66936337854b4f6f921a6c66d2cf2a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.079420 28604 raft_consensus.cc:348] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:31.093065 28290 leader_election.cc:341] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Illegal state: Tablet not RUNNING: SHUTDOWN
I20240503 14:12:31.093338 28290 leader_election.cc:302] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.093940 30616 raft_consensus.cc:2726] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.094763 28614 tablet_bootstrap.cc:654] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.096653 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8087dc9696f46dc9dc8c23a947c7bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.097862 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea649e25074468ea487f77c1a96d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.097631 28604 raft_consensus.cc:374] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.099962 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a28d2b3e5084a1b93f1074d100e397a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.098825 29828 raft_consensus.cc:2258] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.099870 28604 raft_consensus.cc:729] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.102934 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b7aee2f7fe4a37ac7b97250a9e7fe6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:31.102036 28024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.193:45997 (request call id 106) took 1238 ms (client timeout 1000 ms). Trace:
W20240503 14:12:31.105383 28024 rpcz_store.cc:269] 0503 14:12:29.862398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.872965 (+ 10567us) service_pool.cc:225] Handling call
0503 14:12:31.100526 (+1227561us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":42}
I20240503 14:12:31.107848 28604 consensus_queue.cc:260] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:31.113977 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 65177us
I20240503 14:12:31.114498 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a31555fc800142bd849dbd572a194002 (table default.Table_d7b3acfa64194593899187c496e64c66 [id=5d99991da3ec4a0e9ea84e4bb6762014]) successfully deleted
I20240503 14:12:31.114331 29828 ts_tablet_manager.cc:1875] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.116576 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e4f6e3908e420e9fa156ee24b437d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 843f0b87687340558d7c6acbbad2c4d7) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.117298 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.117873 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.118366 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.123615 28427 leader_election.cc:302] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:31.124279 30566 raft_consensus.cc:2781] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:31.124451 30566 raft_consensus.cc:686] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:31.124725 30566 consensus_queue.cc:237] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.122210 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ef2d2d784a834891be22d51c1d9f7ae2" } modify_external_catalogs: true
I20240503 14:12:31.128193 30658 raft_consensus.cc:482] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.129377 30658 raft_consensus.cc:504] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.130450 30658 leader_election.cc:288] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:31.080229 28661 log.cc:1196] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/646f83185d5241268d4126d10561e937
I20240503 14:12:31.140902 28661 ts_tablet_manager.cc:1909] T 646f83185d5241268d4126d10561e937 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:31.142103 28661 tablet_replica.cc:330] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:31.142448 28661 raft_consensus.cc:2229] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.142705 28661 raft_consensus.cc:2258] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.143389 30658 raft_consensus.cc:482] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.143635 30658 raft_consensus.cc:504] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.144546 30658 leader_election.cc:288] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:31.147879 28614 tablet_bootstrap.cc:492] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:31.148602 28614 ts_tablet_manager.cc:1367] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.004s
I20240503 14:12:31.148617 28427 leader_election.cc:302] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:31.150171 30566 raft_consensus.cc:2781] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:31.150343 30566 raft_consensus.cc:686] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:31.150615 30566 consensus_queue.cc:237] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.151885 28614 raft_consensus.cc:348] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.152736 28614 raft_consensus.cc:374] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.153071 28614 raft_consensus.cc:729] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.153895 28614 consensus_queue.cc:260] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.159577 28604 ts_tablet_manager.cc:1398] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.082s user 0.005s sys 0.001s
W20240503 14:12:31.163861 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 73461.9us
I20240503 14:12:31.164013 30658 raft_consensus.cc:2726] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.166479 28661 ts_tablet_manager.cc:1875] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:31.167575 28150 leader_election.cc:341] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.169669 28604 tablet_bootstrap.cc:492] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:31.170454 28349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59945d011b64cba87c854e2ffde8226" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.173556 28600 tablet_bootstrap.cc:492] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.173873 28600 ts_tablet_manager.cc:1367] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.167s user 0.000s sys 0.013s
I20240503 14:12:31.176424 30404 raft_consensus.cc:2258] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.176700 28600 raft_consensus.cc:348] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.176982 28600 raft_consensus.cc:374] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.177193 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.177310 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.177460 28181 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f5103b31ed45b480252dcf94b72f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.177524 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.177055 28600 raft_consensus.cc:729] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.177587 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet ac752d8ab4fd4a218b052f388f416494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.177680 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 007a892ef8504b848a400c05740eaa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 320c526e845e4dc78da345bcb095918f) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.177779 28600 consensus_queue.cc:260] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.178056 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c0540960ac45aabeaf13f888caf238) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.178210 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.178396 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.178519 28600 ts_tablet_manager.cc:1398] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:31.178586 28183 tablet_service.cc:1514] Processing DeleteTablet for tablet b59945d011b64cba87c854e2ffde8226 with delete_type TABLET_DATA_DELETED (replaced by tablet 47965a978a904f2d876690de01f89771) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.178722 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 646f83185d5241268d4126d10561e937 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.178884 28600 tablet_bootstrap.cc:492] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.177932 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet aac51b83abe14ab5b1fb5d627233e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.179414 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet e220336186f648419b186feada6f0124 with delete_type TABLET_DATA_DELETED (replaced by tablet a83532bf2852473a9c51ce7cc78b5d34) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.179631 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.179829 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.177199 30404 ts_tablet_manager.cc:1875] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.180847 28600 tablet_bootstrap.cc:654] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.180361 28614 ts_tablet_manager.cc:1398] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240503 14:12:31.181735 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59945d011b64cba87c854e2ffde8226" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:31.181982 28208 raft_consensus.cc:2454] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:31.182961 30658 raft_consensus.cc:482] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.183250 30658 raft_consensus.cc:504] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.184206 30655 raft_consensus.cc:482] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.184377 30655 raft_consensus.cc:504] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.184962 30655 leader_election.cc:288] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:31.184213 30658 leader_election.cc:288] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:31.189404 28427 leader_election.cc:302] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:31.189767 30446 raft_consensus.cc:2781] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:31.190197 28427 leader_election.cc:302] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:31.190454 30451 consensus_queue.cc:1035] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:31.190956 30404 ts_tablet_manager.cc:1888] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.191090 30404 log.cc:1196] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7d348da459634561985c4ac1710c0bb2
I20240503 14:12:31.191218 30451 raft_consensus.cc:2781] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:31.191293 30451 raft_consensus.cc:482] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:31.191375 30451 raft_consensus.cc:3037] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.190280 28614 tablet_bootstrap.cc:492] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.191464 30404 ts_tablet_manager.cc:1909] T 7d348da459634561985c4ac1710c0bb2 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.192123 30404 tablet_replica.cc:330] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.192353 30404 raft_consensus.cc:2229] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.192518 30404 raft_consensus.cc:2258] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.192623 30616 raft_consensus.cc:482] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.192822 30616 raft_consensus.cc:504] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.192888 30626 tablet.cc:1722] T 7261088abbc641db8d737a48bbac25bd P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:31.193168 30404 ts_tablet_manager.cc:1875] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.193814 30616 leader_election.cc:288] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.196286 30451 raft_consensus.cc:504] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.197189 30451 leader_election.cc:288] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.189930 30446 raft_consensus.cc:482] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:31.200402 30446 raft_consensus.cc:3037] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.203437 28614 tablet_bootstrap.cc:654] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.203311 30446 raft_consensus.cc:504] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.205667 30446 leader_election.cc:288] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.206310 30404 ts_tablet_manager.cc:1888] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.206470 30404 log.cc:1196] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/0c7779f7a9544e17aea7135eab2ccd5f
I20240503 14:12:31.206792 30404 ts_tablet_manager.cc:1909] T 0c7779f7a9544e17aea7135eab2ccd5f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.210372 30404 tablet_replica.cc:330] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.210999 30404 raft_consensus.cc:2229] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.211253 30404 raft_consensus.cc:2258] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.213007 30404 ts_tablet_manager.cc:1875] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.220539 28600 tablet_bootstrap.cc:492] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.220731 28600 ts_tablet_manager.cc:1367] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240503 14:12:31.221554 28604 tablet_bootstrap.cc:654] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.168587 28150 leader_election.cc:302] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:31.223511 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 56092.5us
W20240503 14:12:31.223814 28150 leader_election.cc:341] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.223948 28150 leader_election.cc:302] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.227598 30404 ts_tablet_manager.cc:1888] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.229760 30404 log.cc:1196] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a31555fc800142bd849dbd572a194002
I20240503 14:12:31.230471 30404 ts_tablet_manager.cc:1909] T a31555fc800142bd849dbd572a194002 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.231863 28614 tablet_bootstrap.cc:492] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:31.232069 28614 ts_tablet_manager.cc:1367] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.003s
W20240503 14:12:31.198261 28032 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1906ab252ea349cfa0afbd9f4303efbc consensus state has no leader
I20240503 14:12:31.166760 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 31ddd791d4e14630bfb3b4b1513f76e1 (table default.Table_cf24dd3520bd4373acfd719f2c7737ad [id=e0a3f887fca8477ba19c4ed3738b1955]) successfully deleted
W20240503 14:12:31.235836 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 69340.7us
I20240503 14:12:31.198632 28031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 01482b478dcd45598fa65d14904fe276
I20240503 14:12:31.236819 28031 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33344 (request call id 1252) took 1486 ms. Trace:
I20240503 14:12:31.236860 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7d348da459634561985c4ac1710c0bb2 (table default.Table_ebd8ae7785ef42cca8a6b747de4e0285 [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:31.237149 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 0c7779f7a9544e17aea7135eab2ccd5f (table default.Table_2a170feff0884500b519684ac43a06b0 [id=2d1a26ea051048589afec842f2a31c72]) successfully deleted
I20240503 14:12:31.198911 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 61da48859c4d4a73bb1b9d7c0ede8106
I20240503 14:12:31.237782 28030 catalog_manager.cc:6554] ReplaceTablet: tablet 61da48859c4d4a73bb1b9d7c0ede8106 deleted and replaced by tablet 27b25518570e4f2eb2666d420a9ef39c
I20240503 14:12:31.238302 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_52c51767da7d4d86b088b1bb16ce9623" } new_table_name: "default.Table_df9292b99b3e4128aabcd9c274e3c19a" modify_external_catalogs: true
I20240503 14:12:31.240435 30653 raft_consensus.cc:2726] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.240670 30653 raft_consensus.cc:2726] T 6b94092725fa4956a4c00e3685f9aeaf P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:31.240888 28150 leader_election.cc:341] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.201099 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
I20240503 14:12:31.241556 28026 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 8acbc154c03c47c1b2f9f433fcb2ef9a (table default.Table_bfff85704d88463383cbf94f9b121359 [id=bd1950deb4274d86bf288d501cf5c710]). Expected version 2 got 0
I20240503 14:12:31.242343 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788e4bfa1e414630939bc19da1ede3fb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.243489 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24ab3a5bba04a5092225a981f292222" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.243898 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebec4622ef4950bd1c70451958c59b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.244391 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f81bbdb2ee149c5a446c608034585fe" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:31.246544 28150 leader_election.cc:341] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.246805 28150 leader_election.cc:302] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.247112 28355 raft_consensus.cc:3037] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.201222 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ef2d2d784a834891be22d51c1d9f7ae2" } modify_external_catalogs: true
I20240503 14:12:31.237090 28031 rpcz_store.cc:276] 0503 14:12:29.749779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.135214 (+385435us) service_pool.cc:225] Handling call
0503 14:12:30.332159 (+196945us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x616000032254. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 00007fe12843415d 00007fe128434253 00007fe125165d41 00007fe12cd5234d 00007fe12cedfca6 00007fe123ed1f66 00007fe123ed1c95 00007fe1205fc04e 00007fe1205faf2e 00007fe120601405 00007fe120601d2f
0503 14:12:30.332692 (+ 533us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:30.332827 (+ 135us) catalog_manager.cc:2749] Modifying in-memory table state
0503 14:12:30.332896 (+ 69us) catalog_manager.cc:2758] Locking tablets
0503 14:12:30.862226 (+529330us) catalog_manager.cc:2772] Removing table and tablets from system table
0503 14:12:30.943898 (+ 81672us) catalog_manager.cc:2790] Removing table from by-name map
0503 14:12:31.198450 (+254552us) catalog_manager.cc:2806] Committing in-memory state
0503 14:12:31.198509 (+ 59us) catalog_manager.cc:2810] Aborting table tasks
0503 14:12:31.236723 (+ 38214us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:30.873758 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:30.873903 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:30.873911 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:30.873915 (+ 4us) tablet.cc:660] Decoding operations
0503 14:12:30.874062 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:30.874076 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:30.874082 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0503 14:12:30.874229 (+ 147us) tablet.cc:699] Row locks acquired
0503 14:12:30.874233 (+ 4us) write_op.cc:260] PREPARE: finished
0503 14:12:30.874292 (+ 59us) write_op.cc:270] Start()
0503 14:12:30.874334 (+ 42us) write_op.cc:276] Timestamp: P: 1714745550874246 usec, L: 0
0503 14:12:30.874338 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:30.874602 (+ 264us) log.cc:842] Serialized 731 byte log entry
0503 14:12:30.940884 (+ 66282us) op_driver.cc:444] REPLICATION: finished
0503 14:12:30.943478 (+ 2594us) write_op.cc:301] APPLY: starting
0503 14:12:30.943489 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:30.943519 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:30.943541 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:30.943598 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:30.943623 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0503 14:12:30.943630 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:30.943678 (+ 48us) log.cc:842] Serialized 29 byte log entry
0503 14:12:30.943735 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.943749 (+ 14us) write_op.cc:454] Released schema lock
0503 14:12:30.943754 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:30.943808 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:30.943812 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":452326784,"child_traces":[["op",{"apply.queue_time_us":2565,"apply.run_cpu_time_us":378,"apply.run_wall_time_us":378,"num_ops":2,"prepare.queue_time_us":10620,"prepare.run_cpu_time_us":952,"prepare.run_wall_time_us":1435,"replication_time_us":66534}]]}
I20240503 14:12:31.249264 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b7b3cecd524e40feb9c6251d0403b764" } new_table_name: "default.Table_ab9d419d54b84d1e97ef91d8dc4510e0" modify_external_catalogs: true
I20240503 14:12:31.219461 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_79f99b72ae704c34a5a58eaac219d32e" } modify_external_catalogs: true
I20240503 14:12:31.250650 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788e4bfa1e414630939bc19da1ede3fb" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.218528 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d0e52f75ad7f415bac0142d93ff4aa78" } new_table_name: "default.Table_9dc8c5fcf27e4fddac5ff39852a75d20" modify_external_catalogs: true
I20240503 14:12:31.251149 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f81bbdb2ee149c5a446c608034585fe" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.251652 28343 raft_consensus.cc:3037] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.251958 30404 tablet_replica.cc:330] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.252219 30404 raft_consensus.cc:2229] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.252388 30404 raft_consensus.cc:2258] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.217428 28027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7efeff78f7b04eadbc81cf4b5fb025a1" } modify_external_catalogs: true
I20240503 14:12:31.253237 28027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7efeff78f7b04eadbc81cf4b5fb025a1" } modify_external_catalogs: true
I20240503 14:12:31.253518 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_79f99b72ae704c34a5a58eaac219d32e" } modify_external_catalogs: true
I20240503 14:12:31.253639 30404 ts_tablet_manager.cc:1875] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.254977 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24ab3a5bba04a5092225a981f292222" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.255429 28345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebec4622ef4950bd1c70451958c59b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.221787 28600 raft_consensus.cc:348] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.261440 28600 raft_consensus.cc:374] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.261569 28600 raft_consensus.cc:729] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.261853 28600 consensus_queue.cc:260] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.262513 28600 ts_tablet_manager.cc:1398] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240503 14:12:31.262895 28600 tablet_bootstrap.cc:492] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.265655 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a31555fc800142bd849dbd572a194002 (table default.Table_d7b3acfa64194593899187c496e64c66 [id=5d99991da3ec4a0e9ea84e4bb6762014]) successfully deleted
I20240503 14:12:31.265888 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 01482b478dcd45598fa65d14904fe276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.266177 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 61da48859c4d4a73bb1b9d7c0ede8106 with delete_type TABLET_DATA_DELETED (replaced by tablet 27b25518570e4f2eb2666d420a9ef39c) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:31.268582 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 78391.9us
I20240503 14:12:31.200430 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a88cf5de223f4a6abc21691fea1c4da0
I20240503 14:12:31.272620 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f2d1edcdf50346739221f0a19001dd4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.273604 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2d1edcdf50346739221f0a19001dd4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.277334 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59945d011b64cba87c854e2ffde8226" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:31.277768 28205 raft_consensus.cc:3037] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.280400 30404 ts_tablet_manager.cc:1888] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.280932 28614 raft_consensus.cc:348] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.281186 28614 raft_consensus.cc:374] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.281242 28614 raft_consensus.cc:729] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.281037 28205 raft_consensus.cc:2454] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:31.281508 28614 consensus_queue.cc:260] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.281852 30404 log.cc:1196] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/3dfd6d40a57e4bea973874ff12c0ea3a
I20240503 14:12:31.282114 28614 ts_tablet_manager.cc:1398] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.050s user 0.000s sys 0.000s
I20240503 14:12:31.282449 30632 tablet.cc:1722] T 7261088abbc641db8d737a48bbac25bd P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:31.283447 29828 ts_tablet_manager.cc:1888] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.283584 29828 log.cc:1196] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/007a892ef8504b848a400c05740eaa0c
I20240503 14:12:31.282565 30404 ts_tablet_manager.cc:1909] T 3dfd6d40a57e4bea973874ff12c0ea3a P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.283905 29828 ts_tablet_manager.cc:1909] T 007a892ef8504b848a400c05740eaa0c P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.283993 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac752d8ab4fd4a218b052f388f416494" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:31.284335 28207 raft_consensus.cc:3037] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.284626 29828 tablet_replica.cc:330] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.284864 29828 raft_consensus.cc:2229] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.287921 28614 tablet_bootstrap.cc:492] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.285068 29828 raft_consensus.cc:2258] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.289637 30566 raft_consensus.cc:482] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.291042 29828 ts_tablet_manager.cc:1875] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.291098 30404 tablet_replica.cc:330] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.291298 30404 raft_consensus.cc:2229] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.291492 30404 raft_consensus.cc:2258] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.292106 30404 ts_tablet_manager.cc:1875] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.294260 28614 tablet_bootstrap.cc:654] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.296697 30646 tablet.cc:1722] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240503 14:12:31.289921 30566 raft_consensus.cc:504] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.301103 30566 leader_election.cc:288] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.304289 28661 ts_tablet_manager.cc:1888] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.304560 28661 log.cc:1196] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7d348da459634561985c4ac1710c0bb2
I20240503 14:12:31.304993 28661 ts_tablet_manager.cc:1909] T 7d348da459634561985c4ac1710c0bb2 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:31.306128 28661 tablet_replica.cc:330] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:31.306382 28661 raft_consensus.cc:2229] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:31.306598 28661 pending_rounds.cc:62] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:31.306659 28661 pending_rounds.cc:68] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597772349353984 op_type: NO_OP noop_request { }
I20240503 14:12:31.306797 28661 raft_consensus.cc:2866] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:31.306902 28661 raft_consensus.cc:2258] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.307559 28661 ts_tablet_manager.cc:1875] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.311194 28604 tablet_bootstrap.cc:492] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.311398 28604 ts_tablet_manager.cc:1367] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.145s user 0.004s sys 0.005s
I20240503 14:12:31.312222 28604 raft_consensus.cc:348] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.312448 28604 raft_consensus.cc:374] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.312500 28604 raft_consensus.cc:729] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.312745 28604 consensus_queue.cc:260] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.312844 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 3dfd6d40a57e4bea973874ff12c0ea3a (table default.Table_4eec071da1eb4980b74b08ce5b21765b [id=50c1a152084e4b9690986b223b55e70a]) successfully deleted
I20240503 14:12:31.313477 28604 ts_tablet_manager.cc:1398] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240503 14:12:31.313663 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 65e23e136ac94f9fa4180e15a81122a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e23e136ac94f9fa4180e15a81122a5 already in progress: creating tablet
I20240503 14:12:31.313792 28604 tablet_bootstrap.cc:492] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:31.314319 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 471992c1f7e84ef1953cd85448a492ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 471992c1f7e84ef1953cd85448a492ec already in progress: creating tablet
W20240503 14:12:31.314882 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9cad0a4cdfa8471bbd7066b91e85022e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cad0a4cdfa8471bbd7066b91e85022e already in progress: creating tablet
I20240503 14:12:31.315243 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet a88cf5de223f4a6abc21691fea1c4da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.315533 28604 tablet_bootstrap.cc:654] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.317616 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 530f24322a454f7bb04e818ca865a369
I20240503 14:12:31.318554 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_02b32574c2ac4fdab08b851ecd21084e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.319200 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02b32574c2ac4fdab08b851ecd21084e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:31.319298 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db88ca39eba44740994dd05b526e8506 consensus state has no leader
I20240503 14:12:31.319877 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f6d60f02736f42f1a82698ec52941104"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.320745 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6d60f02736f42f1a82698ec52941104 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.321581 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f783ee62adba49579a0518b240b795bc
I20240503 14:12:31.322432 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1fbfaf1680ed48ef8c60061da7249ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.322978 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fbfaf1680ed48ef8c60061da7249ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.323326 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 9028a6990d0840ffa6849143f6c004db
I20240503 14:12:31.324765 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_ee4900123ee64b90bfc6b5e340346bcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.325510 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee4900123ee64b90bfc6b5e340346bcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.329003 28207 raft_consensus.cc:2454] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:31.331368 28461 tablet_service.cc:1514] Processing DeleteTablet for tablet 01482b478dcd45598fa65d14904fe276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.331681 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 61da48859c4d4a73bb1b9d7c0ede8106 with delete_type TABLET_DATA_DELETED (replaced by tablet 27b25518570e4f2eb2666d420a9ef39c) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.331874 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet a88cf5de223f4a6abc21691fea1c4da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.332731 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b86451bc448428da7485e70e0d54b89" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:31.332980 28205 raft_consensus.cc:2454] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:31.334085 28205 raft_consensus.cc:1261] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 2 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:12:31.336865 30658 raft_consensus.cc:2726] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:31.339569 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 93061.2us
W20240503 14:12:31.339969 28150 leader_election.cc:341] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.340245 28150 leader_election.cc:341] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.340512 28150 leader_election.cc:341] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.340896 28150 leader_election.cc:341] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.341638 30658 raft_consensus.cc:482] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.341782 30658 raft_consensus.cc:504] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.342482 30658 leader_election.cc:288] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:31.343523 28150 leader_election.cc:341] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.343775 28150 leader_election.cc:302] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.343786 30566 consensus_queue.cc:1035] T 86f1ccc26f50432c9ef8789c01ef3519 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:31.344316 30658 raft_consensus.cc:2726] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:31.345038 28150 leader_election.cc:341] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.345060 28427 leader_election.cc:302] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:31.345363 30566 raft_consensus.cc:2781] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:31.345484 30566 raft_consensus.cc:686] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:31.345754 30566 consensus_queue.cc:237] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.348431 28661 ts_tablet_manager.cc:1888] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.353289 28600 tablet_bootstrap.cc:654] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.353363 28661 log.cc:1196] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/3dfd6d40a57e4bea973874ff12c0ea3a
I20240503 14:12:31.353900 28661 ts_tablet_manager.cc:1909] T 3dfd6d40a57e4bea973874ff12c0ea3a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:31.363018 28604 tablet_bootstrap.cc:492] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.363260 28604 ts_tablet_manager.cc:1367] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.001s
I20240503 14:12:31.364322 28427 leader_election.cc:302] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:31.364812 30666 raft_consensus.cc:2781] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:31.365126 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1137) took 1047 ms. Trace:
I20240503 14:12:31.365262 30666 raft_consensus.cc:686] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:31.365681 30666 consensus_queue.cc:237] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.365330 28661 rpcz_store.cc:276] 0503 14:12:30.317861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.317949 (+ 88us) service_pool.cc:225] Handling call
0503 14:12:31.306062 (+988113us) ts_tablet_manager.cc:1120] Deleting tablet 3dfd6d40a57e4bea973874ff12c0ea3a
0503 14:12:31.306073 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.326443 (+ 20370us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.348287 (+ 21844us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.365012 (+ 16725us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5,"tablet-delete.queue_time_us":987994}
I20240503 14:12:31.366591 28661 tablet_replica.cc:330] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
W20240503 14:12:31.369693 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 54291.6us
I20240503 14:12:31.369975 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 646f83185d5241268d4126d10561e937 (table default.Table_4d256c29f9b7400497caee67285a9831 [id=6c9175a58b8544798ce769460dfd9cf5]) successfully deleted
I20240503 14:12:31.370301 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 7d348da459634561985c4ac1710c0bb2 (table default.Table_ebd8ae7785ef42cca8a6b747de4e0285 [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:31.371111 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 530f24322a454f7bb04e818ca865a369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.371843 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet f783ee62adba49579a0518b240b795bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.372627 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 9028a6990d0840ffa6849143f6c004db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.373282 30404 ts_tablet_manager.cc:1888] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.373402 30404 log.cc:1196] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/34238c9859724d97a32f2a1d9315edb1
I20240503 14:12:31.373741 30404 ts_tablet_manager.cc:1909] T 34238c9859724d97a32f2a1d9315edb1 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.375239 30404 tablet_replica.cc:330] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.375505 30404 raft_consensus.cc:2229] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:31.375707 30404 pending_rounds.cc:62] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:31.375752 30404 pending_rounds.cc:68] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597769678561280 op_type: NO_OP noop_request { }
I20240503 14:12:31.375880 30404 raft_consensus.cc:2866] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:31.375967 30404 raft_consensus.cc:2258] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.376184 29828 ts_tablet_manager.cc:1888] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.376367 29828 log.cc:1196] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6de9e159d2ba47039dd6bd207a9cddeb
I20240503 14:12:31.376611 30404 ts_tablet_manager.cc:1875] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.376829 29828 ts_tablet_manager.cc:1909] T 6de9e159d2ba47039dd6bd207a9cddeb P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.377230 28661 raft_consensus.cc:2229] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:31.377583 28661 pending_rounds.cc:62] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:31.377664 28661 pending_rounds.cc:68] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597775718223872 op_type: NO_OP noop_request { }
I20240503 14:12:31.377806 29828 tablet_replica.cc:330] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.377842 28661 raft_consensus.cc:2866] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:31.377961 28661 raft_consensus.cc:2258] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.377991 29828 raft_consensus.cc:2229] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.378122 29828 raft_consensus.cc:2258] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.370105 28427 leader_election.cc:302] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:31.378893 30446 raft_consensus.cc:2781] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:31.379024 30446 raft_consensus.cc:482] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:31.379112 30446 raft_consensus.cc:3037] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.345319 28150 leader_election.cc:302] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.380277 30658 raft_consensus.cc:2726] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:31.380940 28150 leader_election.cc:341] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.381531 28150 leader_election.cc:302] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.381826 30446 raft_consensus.cc:504] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.382122 30658 raft_consensus.cc:2726] T b24ab3a5bba04a5092225a981f292222 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.382539 30446 leader_election.cc:288] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.382783 30446 consensus_queue.cc:1035] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:12:31.382807 28150 leader_election.cc:341] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.382969 28150 leader_election.cc:302] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.383281 30658 raft_consensus.cc:2726] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:31.383769 28427 consensus_peers.cc:485] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31ddd791d4e14630bfb3b4b1513f76e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:31.383816 30404 ts_tablet_manager.cc:1888] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.383921 30404 log.cc:1196] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/cf810e84367f420b928a171790849445
I20240503 14:12:31.384909 30626 tablet.cc:1717] T d832cd30f2414303818d34a0cdf47e60 P d472d9f94d8949cf86284d32a161cebc: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:31.385795 30404 ts_tablet_manager.cc:1909] T cf810e84367f420b928a171790849445 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.386476 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2369dd6d16e4b9681108f2c8574f658" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.388435 30404 tablet_replica.cc:330] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.388780 30404 raft_consensus.cc:2229] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.389010 30404 raft_consensus.cc:2258] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.389652 30404 ts_tablet_manager.cc:1875] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.391835 28661 ts_tablet_manager.cc:1875] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.391953 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2369dd6d16e4b9681108f2c8574f658" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.395862 30404 ts_tablet_manager.cc:1888] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.396109 30404 log.cc:1196] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/5cdc868ddf0a490292eae78ebc8c8ec6
I20240503 14:12:31.396476 30404 ts_tablet_manager.cc:1909] T 5cdc868ddf0a490292eae78ebc8c8ec6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.396971 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 34238c9859724d97a32f2a1d9315edb1 (table default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]) successfully deleted
W20240503 14:12:31.397289 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6fbf20a347f54724a0ce807436863785 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fbf20a347f54724a0ce807436863785 already in progress: creating tablet
W20240503 14:12:31.397498 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet aad397ed76ea4cf8820312ba1362512f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aad397ed76ea4cf8820312ba1362512f already in progress: creating tablet
I20240503 14:12:31.397650 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet cf810e84367f420b928a171790849445 (table default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]) successfully deleted
I20240503 14:12:31.373513 28604 raft_consensus.cc:348] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.398344 28604 raft_consensus.cc:374] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.400413 30404 tablet_replica.cc:330] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.400594 28604 raft_consensus.cc:729] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.400652 30404 raft_consensus.cc:2229] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.400801 30404 raft_consensus.cc:2258] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.401199 28604 consensus_queue.cc:260] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.401405 30404 ts_tablet_manager.cc:1875] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.401916 28604 ts_tablet_manager.cc:1398] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.039s user 0.003s sys 0.000s
I20240503 14:12:31.403091 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b86451bc448428da7485e70e0d54b89" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:31.403309 28212 raft_consensus.cc:3037] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.405997 28212 raft_consensus.cc:2454] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:31.407207 28604 tablet_bootstrap.cc:492] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:31.409127 28604 tablet_bootstrap.cc:654] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.411080 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 3dfd6d40a57e4bea973874ff12c0ea3a (table default.Table_4eec071da1eb4980b74b08ce5b21765b [id=50c1a152084e4b9690986b223b55e70a]) successfully deleted
I20240503 14:12:31.411465 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 5cdc868ddf0a490292eae78ebc8c8ec6 (table default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3 [id=ca041900898046468ff2eec0d0fadd85]) successfully deleted
I20240503 14:12:31.414089 28614 tablet_bootstrap.cc:492] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:31.414332 28614 ts_tablet_manager.cc:1367] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.132s user 0.002s sys 0.007s
I20240503 14:12:31.415058 28614 raft_consensus.cc:348] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.415279 28614 raft_consensus.cc:374] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.415338 28614 raft_consensus.cc:729] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.415572 28614 consensus_queue.cc:260] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.416293 30632 tablet.cc:1717] T d832cd30f2414303818d34a0cdf47e60 P a64bc15d65d84ad5a005960a9a84df9c: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:31.416685 29828 ts_tablet_manager.cc:1875] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.418555 28600 tablet_bootstrap.cc:492] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.418808 28600 ts_tablet_manager.cc:1367] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.156s user 0.005s sys 0.000s
I20240503 14:12:31.409972 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 530f24322a454f7bb04e818ca865a369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.410207 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet f783ee62adba49579a0518b240b795bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.410333 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 9028a6990d0840ffa6849143f6c004db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.416036 28427 leader_election.cc:302] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:31.420408 30666 raft_consensus.cc:2781] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:31.420420 28600 raft_consensus.cc:348] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.421001 28600 raft_consensus.cc:374] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.421180 28600 raft_consensus.cc:729] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.421644 28600 consensus_queue.cc:260] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.422787 28600 ts_tablet_manager.cc:1398] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:31.423245 30666 raft_consensus.cc:686] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:31.423432 28600 tablet_bootstrap.cc:492] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.423548 30666 consensus_queue.cc:237] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.428697 28614 ts_tablet_manager.cc:1398] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240503 14:12:31.429200 28614 tablet_bootstrap.cc:492] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.431818 28614 tablet_bootstrap.cc:654] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.432741 30404 ts_tablet_manager.cc:1888] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.432976 30404 log.cc:1196] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/d6f5103b31ed45b480252dcf94b72f9c
I20240503 14:12:31.433468 30404 ts_tablet_manager.cc:1909] T d6f5103b31ed45b480252dcf94b72f9c P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.434422 30335 raft_consensus.cc:482] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.434579 30335 raft_consensus.cc:504] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.435235 30335 leader_election.cc:288] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:31.436739 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet d6f5103b31ed45b480252dcf94b72f9c (table default.Table_41a322f940d942adab752b8952a9f4a4 [id=18b53c1cdfb4468296f5bad79e15fb01]) successfully deleted
W20240503 14:12:31.440358 28150 leader_election.cc:341] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.440635 28150 leader_election.cc:341] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.440734 28150 leader_election.cc:302] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:31.441493 28427 leader_election.cc:341] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: d7c79ffee3954b5c870f4a8aa6379517
W20240503 14:12:31.442281 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6de9e159d2ba47039dd6bd207a9cddeb because the tablet was not found. No further retry: Not found: Tablet not found: 6de9e159d2ba47039dd6bd207a9cddeb
I20240503 14:12:31.442672 30404 tablet_replica.cc:330] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.446584 30658 raft_consensus.cc:2726] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.442970 30404 raft_consensus.cc:2229] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.460704 30404 raft_consensus.cc:2258] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.461565 30404 ts_tablet_manager.cc:1875] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.463105 28600 tablet_bootstrap.cc:654] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.475574 30658 raft_consensus.cc:482] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.475772 30658 raft_consensus.cc:504] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.476547 30658 leader_election.cc:288] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
W20240503 14:12:31.477047 28427 consensus_peers.cc:485] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c7779f7a9544e17aea7135eab2ccd5f. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:31.477406 28427 consensus_peers.cc:485] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34238c9859724d97a32f2a1d9315edb1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:31.481698 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a0ac35f174d0485f89df474c6f75f7ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.482421 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0ac35f174d0485f89df474c6f75f7ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.482388 30404 ts_tablet_manager.cc:1888] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.482641 30404 log.cc:1196] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/ac752d8ab4fd4a218b052f388f416494
I20240503 14:12:31.483171 30404 ts_tablet_manager.cc:1909] T ac752d8ab4fd4a218b052f388f416494 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.483131 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8ce861018efe45e58f9c7ed20c85a6d7" } new_table_name: "default.Table_4a23f2d4723c4ed79f3891fe2253614f" modify_external_catalogs: true
W20240503 14:12:31.483839 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58fe166ca0634e5eb1ae4cba526f494c consensus state has no leader
I20240503 14:12:31.484123 28030 master_service.cc:946] ReplaceTablet: received request to replace tablet ceec6992b0474706b8b3766518b0c274 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:31.484789 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 9ac2833a0bb045dcb2824468d936ffe4 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:31.484843 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet ac752d8ab4fd4a218b052f388f416494 (table default.Table_c7e5a76bf36545a5b53fc3d60512a521 [id=e6889b2bf6214cabba6669c414c22f7a]) successfully deleted
I20240503 14:12:31.484997 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 77f6fb009bdd4f70ba90e7b5f972cea9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:31.485414 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_9099811cf7874977804669fff91540ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.485705 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 007a892ef8504b848a400c05740eaa0c because the tablet was not found. No further retry: Not found: Tablet not found: 007a892ef8504b848a400c05740eaa0c
W20240503 14:12:31.486001 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9099811cf7874977804669fff91540ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.485958 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2742208c2e8f4596952e282b1b29c727"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.486431 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet d7c79ffee3954b5c870f4a8aa6379517 because the tablet was not found. No further retry: Not found: Tablet not found: d7c79ffee3954b5c870f4a8aa6379517
W20240503 14:12:31.486572 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2742208c2e8f4596952e282b1b29c727 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:31.487069 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 6696a237c3dd428097e2ebdf24afd63f because the tablet was not found. No further retry: Not found: Tablet not found: 6696a237c3dd428097e2ebdf24afd63f
I20240503 14:12:31.487388 30404 tablet_replica.cc:330] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.487617 30404 raft_consensus.cc:2229] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.487790 30404 raft_consensus.cc:2258] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.488435 30404 ts_tablet_manager.cc:1875] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.490249 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b4c9a47c0458ba188c0cf62f3fe48" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.490878 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b4c9a47c0458ba188c0cf62f3fe48" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.485433 28027 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b86451bc448428da7485e70e0d54b89 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:31.497119 30404 ts_tablet_manager.cc:1888] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.497329 30404 log.cc:1196] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/996bb3a7edfc4f5a8a3a39ea6fcfe5cd
I20240503 14:12:31.497650 30404 ts_tablet_manager.cc:1909] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.498871 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd (table default.Table_8d1519e30ef54d12ab71e31903605531 [id=fa291ee5c8f54e89bcef6755f4ff3f0a]) successfully deleted
I20240503 14:12:31.499482 30404 tablet_replica.cc:330] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
W20240503 14:12:31.500180 28150 leader_election.cc:341] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.500427 28150 leader_election.cc:341] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.500504 28150 leader_election.cc:302] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.500761 30658 raft_consensus.cc:2726] T 725b4c9a47c0458ba188c0cf62f3fe48 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.499737 30404 raft_consensus.cc:2229] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.504149 30404 raft_consensus.cc:2258] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.504931 30404 ts_tablet_manager.cc:1875] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:31.512056 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fb570eb39d54f2cac41d4f77b18abc7 consensus state has no leader
I20240503 14:12:31.512708 30404 ts_tablet_manager.cc:1888] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.512642 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_e7c848ebbd6a4038a0404bddcb18f521"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:31.512691 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_2f4bc37491894d27aae1c52cf3d365e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:31.513125 30404 log.cc:1196] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/b59945d011b64cba87c854e2ffde8226
W20240503 14:12:31.513311 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7c848ebbd6a4038a0404bddcb18f521 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:31.513319 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f4bc37491894d27aae1c52cf3d365e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.513628 30404 ts_tablet_manager.cc:1909] T b59945d011b64cba87c854e2ffde8226 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.516988 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_8a699c7b01a04e4aad97227bb13be53a" } new_table_name: "default.Table_89dc20edab5b4dad89adbe39e24a6358" modify_external_catalogs: true
I20240503 14:12:31.517590 28033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2bc25daca38a439a9c3275f9cb988224" } modify_external_catalogs: true
I20240503 14:12:31.517890 28033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2bc25daca38a439a9c3275f9cb988224" } modify_external_catalogs: true
I20240503 14:12:31.519863 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet b59945d011b64cba87c854e2ffde8226 (table default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]) successfully deleted
W20240503 14:12:31.520793 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 646f83185d5241268d4126d10561e937 because the tablet was not found. No further retry: Not found: Tablet not found: 646f83185d5241268d4126d10561e937
I20240503 14:12:31.521158 30404 tablet_replica.cc:330] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.521453 30404 raft_consensus.cc:2229] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.521665 28614 tablet_bootstrap.cc:492] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:31.521697 30404 raft_consensus.cc:2258] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.521832 28614 ts_tablet_manager.cc:1367] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20240503 14:12:31.522522 30404 ts_tablet_manager.cc:1875] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.523380 28614 raft_consensus.cc:348] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.523873 28614 raft_consensus.cc:374] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.524024 28614 raft_consensus.cc:729] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.525108 28614 consensus_queue.cc:260] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.526405 28614 ts_tablet_manager.cc:1398] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:31.528182 28604 tablet_bootstrap.cc:492] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.530078 28604 ts_tablet_manager.cc:1367] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.123s user 0.017s sys 0.011s
I20240503 14:12:31.533748 30404 ts_tablet_manager.cc:1888] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.534273 28604 raft_consensus.cc:348] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.534606 28604 raft_consensus.cc:374] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.534695 28604 raft_consensus.cc:729] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.534994 28604 consensus_queue.cc:260] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.538170 28604 ts_tablet_manager.cc:1398] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:31.538645 28604 tablet_bootstrap.cc:492] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:31.539505 28661 ts_tablet_manager.cc:1888] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.539788 28661 log.cc:1196] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6696a237c3dd428097e2ebdf24afd63f
I20240503 14:12:31.540467 28604 tablet_bootstrap.cc:654] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.540733 28661 ts_tablet_manager.cc:1909] T 6696a237c3dd428097e2ebdf24afd63f P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:31.543727 30404 log.cc:1196] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/aac51b83abe14ab5b1fb5d627233e58e
I20240503 14:12:31.544237 30404 ts_tablet_manager.cc:1909] T aac51b83abe14ab5b1fb5d627233e58e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.545403 30404 tablet_replica.cc:330] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.545754 30404 raft_consensus.cc:2229] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.546403 30404 raft_consensus.cc:2258] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.547128 30404 ts_tablet_manager.cc:1875] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.547370 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet aac51b83abe14ab5b1fb5d627233e58e (table default.Table_618386316f9b46f9a7770c3f998769f5 [id=ffcb298fb844406881d9a8812e24e896]) successfully deleted
I20240503 14:12:31.548168 29828 ts_tablet_manager.cc:1888] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.548364 29828 log.cc:1196] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/49445f0f1dd1451c9ba066e61273f2f1
I20240503 14:12:31.548796 29828 ts_tablet_manager.cc:1909] T 49445f0f1dd1451c9ba066e61273f2f1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.549739 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1122) took 1137 ms. Trace:
I20240503 14:12:31.549863 29828 rpcz_store.cc:276] 0503 14:12:30.412203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.413178 (+ 975us) service_pool.cc:225] Handling call
0503 14:12:31.377746 (+964568us) ts_tablet_manager.cc:1120] Deleting tablet 49445f0f1dd1451c9ba066e61273f2f1
0503 14:12:31.377755 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.439486 (+ 61731us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.509266 (+ 69780us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.549711 (+ 40445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":964416}
I20240503 14:12:31.550251 29828 tablet_replica.cc:330] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.551256 29828 raft_consensus.cc:2229] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.551450 29828 raft_consensus.cc:2258] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.552034 29828 ts_tablet_manager.cc:1875] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.552413 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_da6c5739b62c43758e9e53182b790622" } modify_external_catalogs: true
W20240503 14:12:31.552556 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5eaef52887064ff28c5d0fbbb4090d8f consensus state has no leader
I20240503 14:12:31.552620 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_da6c5739b62c43758e9e53182b790622" } modify_external_catalogs: true
I20240503 14:12:31.552930 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a0ac35f174d0485f89df474c6f75f7ed" } modify_external_catalogs: true
I20240503 14:12:31.553090 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a0ac35f174d0485f89df474c6f75f7ed" } modify_external_catalogs: true
I20240503 14:12:31.555619 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d5c70a4285c24a50b556481cf3793c19" } new_table_name: "default.Table_6530fb7e21e24e76af1b60c4c6cbe6c7" modify_external_catalogs: true
I20240503 14:12:31.558537 30404 ts_tablet_manager.cc:1888] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.560999 30404 log.cc:1196] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/e220336186f648419b186feada6f0124
I20240503 14:12:31.561601 30404 ts_tablet_manager.cc:1909] T e220336186f648419b186feada6f0124 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.562672 28600 tablet_bootstrap.cc:492] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.562840 28600 ts_tablet_manager.cc:1367] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.000s
I20240503 14:12:31.563690 28600 raft_consensus.cc:348] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.563899 28600 raft_consensus.cc:374] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.563953 28600 raft_consensus.cc:729] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.564283 28600 consensus_queue.cc:260] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.566583 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 989ea2f962a7492d8c32e4becec1457d
I20240503 14:12:31.567423 28029 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 5a1e6dc950e64d598cb5f8b333c9086b
I20240503 14:12:31.567484 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fde27a1501fd4d958d10183826b7ec3d" } modify_external_catalogs: true
I20240503 14:12:31.567687 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_fde27a1501fd4d958d10183826b7ec3d" } modify_external_catalogs: true
I20240503 14:12:31.567726 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_38b33f45a641422ba567e4bbadaa6e23" } modify_external_catalogs: true
I20240503 14:12:31.567888 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_38b33f45a641422ba567e4bbadaa6e23" } modify_external_catalogs: true
I20240503 14:12:31.569201 28600 ts_tablet_manager.cc:1398] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:31.569555 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet 989ea2f962a7492d8c32e4becec1457d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.570459 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet e220336186f648419b186feada6f0124 (table default.Table_a7060c6b06e14403ad4ab81fc6e5c8cd [id=ffbda772b5ce4606914f5fecfff4a335]) successfully deleted
I20240503 14:12:31.571770 30335 raft_consensus.cc:482] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.571928 30335 raft_consensus.cc:504] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.572983 30335 leader_election.cc:288] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.573761 28614 tablet_bootstrap.cc:492] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.575623 28614 tablet_bootstrap.cc:654] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.581601 28197 tablet_service.cc:1514] Processing DeleteTablet for tablet 989ea2f962a7492d8c32e4becec1457d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:31.582634 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbc90d9b37bc4c57af3d29ce0f7da287 consensus state has no leader
I20240503 14:12:31.583076 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet aa3ce05b422a4166bfac468acb84e4cc from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:31.594988 29828 ts_tablet_manager.cc:1888] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.595192 29828 log.cc:1196] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/b476e87faea1415d95ae014a93fa22af
I20240503 14:12:31.595535 29828 ts_tablet_manager.cc:1909] T b476e87faea1415d95ae014a93fa22af P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.598445 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1135) took 1280 ms. Trace:
I20240503 14:12:31.598610 28661 rpcz_store.cc:276] 0503 14:12:30.317673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.318090 (+ 417us) service_pool.cc:225] Handling call
0503 14:12:31.366449 (+1048359us) ts_tablet_manager.cc:1120] Deleting tablet 6696a237c3dd428097e2ebdf24afd63f
0503 14:12:31.366461 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.395613 (+ 29152us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.539377 (+143764us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.598397 (+ 59020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048251}
I20240503 14:12:31.599117 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6696a237c3dd428097e2ebdf24afd63f (table default.Table_43c9760cf3b54d68b218cafe8393cda2 [id=821a0387d6f44024b05959350fd53b63]) successfully deleted
I20240503 14:12:31.599229 28661 tablet_replica.cc:330] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:31.599463 28661 raft_consensus.cc:2229] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:31.599853 28661 raft_consensus.cc:2258] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.600631 28661 ts_tablet_manager.cc:1875] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.605433 30638 raft_consensus.cc:482] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.605688 30638 raft_consensus.cc:504] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.606524 30638 leader_election.cc:288] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:31.607645 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6a58db50f849a8be729c10dbe03e76" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.608258 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6a58db50f849a8be729c10dbe03e76" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:31.609090 28150 leader_election.cc:341] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.609359 28150 leader_election.cc:341] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.609445 28150 leader_election.cc:302] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.609622 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1125) took 1197 ms. Trace:
I20240503 14:12:31.609723 30638 raft_consensus.cc:2726] T 0a6a58db50f849a8be729c10dbe03e76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.609735 29828 rpcz_store.cc:276] 0503 14:12:30.412422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.413357 (+ 935us) service_pool.cc:225] Handling call
0503 14:12:31.550191 (+1136834us) ts_tablet_manager.cc:1120] Deleting tablet b476e87faea1415d95ae014a93fa22af
0503 14:12:31.550203 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.583626 (+ 33423us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.594850 (+ 11224us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.609596 (+ 14746us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136718}
I20240503 14:12:31.610158 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1126) took 1197 ms. Trace:
I20240503 14:12:31.610246 29828 rpcz_store.cc:276] 0503 14:12:30.412486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.413491 (+ 1005us) service_pool.cc:225] Handling call
0503 14:12:31.610039 (+1196548us) ts_tablet_manager.cc:1120] Deleting tablet 2cad43f3c6094253ba87d0ed3fc29fd9
0503 14:12:31.610050 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.610141 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196426}
I20240503 14:12:31.610515 29828 tablet_replica.cc:330] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.610721 29828 raft_consensus.cc:2229] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.610894 29828 raft_consensus.cc:2258] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.611466 29828 ts_tablet_manager.cc:1875] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.613096 30335 raft_consensus.cc:482] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.613262 30335 raft_consensus.cc:504] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.613965 30335 leader_election.cc:288] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.614573 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fe462c9ee54a27b9714f6457036d76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:31.614665 28600 tablet_bootstrap.cc:492] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.614799 28208 raft_consensus.cc:2454] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:31.616003 28427 leader_election.cc:302] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:31.616423 28427 leader_election.cc:341] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: e220336186f648419b186feada6f0124
I20240503 14:12:31.616489 30335 raft_consensus.cc:2781] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:31.616551 28600 tablet_bootstrap.cc:654] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.616554 30335 raft_consensus.cc:482] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:31.616616 30335 raft_consensus.cc:3037] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.619356 30335 raft_consensus.cc:504] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:31.620055 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a035a5249f8348b591b27ba3fa7ae43a because the tablet was not found. No further retry: Not found: Tablet not found: a035a5249f8348b591b27ba3fa7ae43a
I20240503 14:12:31.620817 30404 tablet_replica.cc:330] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.621059 30404 raft_consensus.cc:2229] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.621230 30404 raft_consensus.cc:2258] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.621311 29828 ts_tablet_manager.cc:1888] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.621445 29828 log.cc:1196] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/11e14e8b5f4046029af4673a277112d9
I20240503 14:12:31.621739 29828 ts_tablet_manager.cc:1909] T 11e14e8b5f4046029af4673a277112d9 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.621843 30404 ts_tablet_manager.cc:1875] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.623337 30335 leader_election.cc:288] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.625093 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fe462c9ee54a27b9714f6457036d76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:31.625324 28216 raft_consensus.cc:3037] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.628162 28216 raft_consensus.cc:2454] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:31.629078 28427 leader_election.cc:302] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:31.630231 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1123) took 1217 ms. Trace:
I20240503 14:12:31.630558 30666 raft_consensus.cc:2781] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:31.630687 30666 raft_consensus.cc:686] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:31.630962 30666 consensus_queue.cc:237] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.633529 28540 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:31.630337 29828 rpcz_store.cc:276] 0503 14:12:30.412283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.414794 (+ 2511us) service_pool.cc:225] Handling call
0503 14:12:31.610465 (+1195671us) ts_tablet_manager.cc:1120] Deleting tablet 11e14e8b5f4046029af4673a277112d9
0503 14:12:31.610475 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.617629 (+ 7154us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.621183 (+ 3554us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.630196 (+ 9013us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195443}
I20240503 14:12:31.640687 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1130) took 1021 ms. Trace:
I20240503 14:12:31.640815 29828 rpcz_store.cc:276] 0503 14:12:30.619380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.619466 (+ 86us) service_pool.cc:225] Handling call
0503 14:12:31.640525 (+1021059us) ts_tablet_manager.cc:1120] Deleting tablet 65e23e136ac94f9fa4180e15a81122a5
0503 14:12:31.640535 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.640658 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020728}
I20240503 14:12:31.641069 28614 tablet_bootstrap.cc:492] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:31.641234 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1131) took 1021 ms. Trace:
I20240503 14:12:31.641264 28614 ts_tablet_manager.cc:1367] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.068s user 0.000s sys 0.004s
I20240503 14:12:31.641688 30404 ts_tablet_manager.cc:1888] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.641968 30404 log.cc:1196] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/9b21cd1262d94d1c994b29c44f86cb5e
I20240503 14:12:31.642405 30404 ts_tablet_manager.cc:1909] T 9b21cd1262d94d1c994b29c44f86cb5e P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.641307 29828 rpcz_store.cc:276] 0503 14:12:30.619505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.619701 (+ 196us) service_pool.cc:225] Handling call
0503 14:12:31.641073 (+1021372us) ts_tablet_manager.cc:1120] Deleting tablet 471992c1f7e84ef1953cd85448a492ec
0503 14:12:31.641082 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.641218 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1021170}
I20240503 14:12:31.643522 29828 tablet_replica.cc:330] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.643734 29828 raft_consensus.cc:2229] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.643926 29828 raft_consensus.cc:2258] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.644556 29828 ts_tablet_manager.cc:1875] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.644670 28614 raft_consensus.cc:348] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.644873 28614 raft_consensus.cc:374] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.644940 28614 raft_consensus.cc:729] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.645205 28614 consensus_queue.cc:260] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.645363 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 9b21cd1262d94d1c994b29c44f86cb5e (table default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]) successfully deleted
W20240503 14:12:31.646309 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 01482b478dcd45598fa65d14904fe276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01482b478dcd45598fa65d14904fe276 already in progress: creating tablet
W20240503 14:12:31.647029 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 61da48859c4d4a73bb1b9d7c0ede8106 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61da48859c4d4a73bb1b9d7c0ede8106 already in progress: creating tablet
W20240503 14:12:31.647445 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a88cf5de223f4a6abc21691fea1c4da0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88cf5de223f4a6abc21691fea1c4da0 already in progress: creating tablet
W20240503 14:12:31.647841 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 530f24322a454f7bb04e818ca865a369 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 530f24322a454f7bb04e818ca865a369 already in progress: creating tablet
I20240503 14:12:31.648030 28614 ts_tablet_manager.cc:1398] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
W20240503 14:12:31.648121 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f783ee62adba49579a0518b240b795bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f783ee62adba49579a0518b240b795bc already in progress: creating tablet
W20240503 14:12:31.648342 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 9028a6990d0840ffa6849143f6c004db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9028a6990d0840ffa6849143f6c004db already in progress: creating tablet
I20240503 14:12:31.648432 28614 tablet_bootstrap.cc:492] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.650043 30404 tablet_replica.cc:330] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:31.650096 28614 tablet_bootstrap.cc:654] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.650367 30404 raft_consensus.cc:2229] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.650593 30404 raft_consensus.cc:2258] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.651459 30404 ts_tablet_manager.cc:1875] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.652974 28604 tablet_bootstrap.cc:492] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.653185 28604 ts_tablet_manager.cc:1367] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.115s user 0.017s sys 0.033s
I20240503 14:12:31.654074 28604 raft_consensus.cc:348] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.654390 28604 raft_consensus.cc:374] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.654498 28604 raft_consensus.cc:729] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.654812 28604 consensus_queue.cc:260] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.655822 28604 ts_tablet_manager.cc:1398] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:31.656180 28604 tablet_bootstrap.cc:492] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:31.657763 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 58fe166ca0634e5eb1ae4cba526f494c (table default.Table_df9292b99b3e4128aabcd9c274e3c19a [id=1a8e59318503488d8bf4552ed9eb8364], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 58fe166ca0634e5eb1ae4cba526f494c consensus state has no leader
I20240503 14:12:31.659061 28600 tablet_bootstrap.cc:492] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.659229 28600 ts_tablet_manager.cc:1367] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240503 14:12:31.661757 30638 raft_consensus.cc:482] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.661912 30638 raft_consensus.cc:504] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.662575 30638 leader_election.cc:288] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:31.659117 28604 tablet_bootstrap.cc:654] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.664114 28600 raft_consensus.cc:348] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.664428 28600 raft_consensus.cc:374] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.664549 28600 raft_consensus.cc:729] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.665021 28600 consensus_queue.cc:260] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.665498 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fa067359a640eeba8c80d08371904e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.666011 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fa067359a640eeba8c80d08371904e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.666200 28600 ts_tablet_manager.cc:1398] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
W20240503 14:12:31.666663 28150 leader_election.cc:341] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.666891 28150 leader_election.cc:341] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.666973 28150 leader_election.cc:302] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.667198 30638 raft_consensus.cc:2726] T 34fa067359a640eeba8c80d08371904e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.667666 30404 ts_tablet_manager.cc:1888] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.668781 30404 log.cc:1196] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/989ea2f962a7492d8c32e4becec1457d
I20240503 14:12:31.669353 30404 ts_tablet_manager.cc:1909] T 989ea2f962a7492d8c32e4becec1457d P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:31.673152 28604 tablet_bootstrap.cc:492] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.673430 28604 ts_tablet_manager.cc:1367] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240503 14:12:31.675648 28604 raft_consensus.cc:348] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.675987 28604 raft_consensus.cc:374] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.676138 28604 raft_consensus.cc:729] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.676450 28604 consensus_queue.cc:260] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.677264 28604 ts_tablet_manager.cc:1398] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:31.680356 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 9ac2833a0bb045dcb2824468d936ffe4 (table default.Table_472814febc0548599d68e796d57f48a1 [id=7128b9ce63544947ab63aa6d2490c691]) successfully deleted
W20240503 14:12:31.681718 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 62e4f6e3908e420e9fa156ee24b437d6 because the tablet was not found. No further retry: Not found: Tablet not found: 62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:31.686444 28604 tablet_bootstrap.cc:492] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:31.687981 28600 tablet_bootstrap.cc:492] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.692922 28028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_521b7e40755f4748ad55fc13b690b642" } new_table_name: "default.Table_644d3999ff4140198ab4fa385ffcef55" modify_external_catalogs: true
I20240503 14:12:31.698616 29828 ts_tablet_manager.cc:1888] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.698781 29828 log.cc:1196] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/34238c9859724d97a32f2a1d9315edb1
I20240503 14:12:31.698917 28263 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
W20240503 14:12:31.704811 28028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31721f852044499b84ad17c4b31964a4 consensus state has no leader
I20240503 14:12:31.705318 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_fc769de006724067ba44a724436e6a35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.705963 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc769de006724067ba44a724436e6a35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.706971 28600 tablet_bootstrap.cc:654] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.707471 28614 tablet_bootstrap.cc:492] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:31.707617 28614 ts_tablet_manager.cc:1367] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.003s
I20240503 14:12:31.708364 28614 raft_consensus.cc:348] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.708552 28614 raft_consensus.cc:374] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.708606 28614 raft_consensus.cc:729] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.708808 28614 consensus_queue.cc:260] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.710709 28614 ts_tablet_manager.cc:1398] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:31.712191 28614 tablet_bootstrap.cc:492] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.717555 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_a5bf58872e734741a8443b8a1a3f5424"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.720703 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5bf58872e734741a8443b8a1a3f5424 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.719561 28614 tablet_bootstrap.cc:654] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.727200 28604 tablet_bootstrap.cc:654] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.726263 29828 ts_tablet_manager.cc:1909] T 34238c9859724d97a32f2a1d9315edb1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.727521 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 989ea2f962a7492d8c32e4becec1457d (table default.Table_da6c5739b62c43758e9e53182b790622 [id=6b342f8063bb4c2db3d8f63359afa1a8]) successfully deleted
I20240503 14:12:31.727869 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1133) took 1007 ms. Trace:
I20240503 14:12:31.727967 29828 rpcz_store.cc:276] 0503 14:12:30.720801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.720871 (+ 70us) service_pool.cc:225] Handling call
0503 14:12:31.643482 (+922611us) ts_tablet_manager.cc:1120] Deleting tablet 34238c9859724d97a32f2a1d9315edb1
0503 14:12:31.643489 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.679615 (+ 36126us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.698486 (+ 18871us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.727840 (+ 29354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":922421}
I20240503 14:12:31.728451 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1134) took 1007 ms. Trace:
I20240503 14:12:31.728531 29828 rpcz_store.cc:276] 0503 14:12:30.721297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.721354 (+ 57us) service_pool.cc:225] Handling call
0503 14:12:31.728310 (+1006956us) ts_tablet_manager.cc:1120] Deleting tablet 6fbf20a347f54724a0ce807436863785
0503 14:12:31.728320 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.728432 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006752}
I20240503 14:12:31.728925 29828 tablet_replica.cc:330] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.729110 29828 raft_consensus.cc:2229] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.729282 29828 raft_consensus.cc:2258] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.731042 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
I20240503 14:12:31.731381 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_ec97037ea5a849bfb48a72d6771fbf1f [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 2 got 0
I20240503 14:12:31.731642 28025 catalog_manager.cc:5484] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 1 to 2, leader changed from a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193) to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 2 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:31.732960 28661 ts_tablet_manager.cc:1888] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.733110 28661 log.cc:1196] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/a035a5249f8348b591b27ba3fa7ae43a
I20240503 14:12:31.733470 28661 ts_tablet_manager.cc:1909] T a035a5249f8348b591b27ba3fa7ae43a P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:31.733731 29828 ts_tablet_manager.cc:1875] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.734839 30638 raft_consensus.cc:482] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.734993 30638 raft_consensus.cc:504] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.735805 30638 leader_election.cc:288] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:31.736626 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99aa7e59c45c45c7bc1f726c97426b89" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.736819 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59945d011b64cba87c854e2ffde8226" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:31.737118 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99aa7e59c45c45c7bc1f726c97426b89" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:31.738354 28150 leader_election.cc:341] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.738607 28150 leader_election.cc:341] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.738714 28150 leader_election.cc:302] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.738957 30638 raft_consensus.cc:2726] T 99aa7e59c45c45c7bc1f726c97426b89 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.739442 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:31.740471 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:31.741425 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac752d8ab4fd4a218b052f388f416494" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:31.741650 28355 raft_consensus.cc:3037] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.744776 28355 raft_consensus.cc:2454] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:31.749156 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b86451bc448428da7485e70e0d54b89" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.752461 28604 tablet_bootstrap.cc:492] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.752530 28600 tablet_bootstrap.cc:492] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.752637 28604 ts_tablet_manager.cc:1367] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.005s
I20240503 14:12:31.752657 28600 ts_tablet_manager.cc:1367] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.010s
I20240503 14:12:31.752415 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b86451bc448428da7485e70e0d54b89" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:31.753435 30616 raft_consensus.cc:482] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.753643 28355 raft_consensus.cc:1261] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Refusing update from remote peer d472d9f94d8949cf86284d32a161cebc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:12:31.753592 30616 raft_consensus.cc:504] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.753665 28604 raft_consensus.cc:348] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.753849 28604 raft_consensus.cc:374] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.753899 28604 raft_consensus.cc:729] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.754326 30616 leader_election.cc:288] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.754281 28604 consensus_queue.cc:260] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.755127 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 201cbe667ff640cf93e24f4bfcf113e3 (DEFAULT_TABLE table=default.Table_1f1afeacc77b4bf4af1a39ce67f957b3 [id=8dd887797f0f4d2db696c41352a828c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.755267 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c79ffee3954b5c870f4a8aa6379517" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.755384 28600 raft_consensus.cc:348] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.755568 28600 raft_consensus.cc:374] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.755589 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201cbe667ff640cf93e24f4bfcf113e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.755623 28600 raft_consensus.cc:729] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.756328 28355 raft_consensus.cc:3037] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:31.755870 28600 consensus_queue.cc:260] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.756992 28600 ts_tablet_manager.cc:1398] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:31.757304 28600 tablet_bootstrap.cc:492] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.759321 28600 tablet_bootstrap.cc:654] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.760727 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 201cbe667ff640cf93e24f4bfcf113e3 (DEFAULT_TABLE table=default.Table_1f1afeacc77b4bf4af1a39ce67f957b3 [id=8dd887797f0f4d2db696c41352a828c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.761183 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201cbe667ff640cf93e24f4bfcf113e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.761780 28604 ts_tablet_manager.cc:1398] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:12:31.764430 28604 tablet_bootstrap.cc:492] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:31.766515 28604 tablet_bootstrap.cc:654] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.766850 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ad9baf131a1948d8ae5fae4c706aa726
I20240503 14:12:31.767341 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ceec6992b0474706b8b3766518b0c274
I20240503 14:12:31.767745 28024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_97a23538f3064d93a85e1b1218f01e9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:31.767984 28033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 328bb280f36d4c79b2795c10df2b836b
I20240503 14:12:31.767760 28030 catalog_manager.cc:6554] ReplaceTablet: tablet ceec6992b0474706b8b3766518b0c274 deleted and replaced by tablet a58c1c4aba5d4d5c9798e0ce0ee12839
W20240503 14:12:31.768429 28024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97a23538f3064d93a85e1b1218f01e9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.768504 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3e8ffe332027447da16c340fb4fb6f5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.768990 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e8ffe332027447da16c340fb4fb6f5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.770797 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet ac0813ac39364a268dd7df98d32403c6
I20240503 14:12:31.771667 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_c184e04cc722407c84daac50b15af9e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.772303 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c184e04cc722407c84daac50b15af9e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.774940 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1138) took 1457 ms. Trace:
I20240503 14:12:31.775279 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_1e60a3851c8f4b49985e97e1d6096ac0" } new_table_name: "default.Table_7b3cfdaae8184a42ad15a6ffcacb34da" modify_external_catalogs: true
I20240503 14:12:31.776369 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fe462c9ee54a27b9714f6457036d76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.779412 28600 tablet_bootstrap.cc:492] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.780319 28600 ts_tablet_manager.cc:1367] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240503 14:12:31.781519 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9baf131a1948d8ae5fae4c706aa726 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.781909 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet ceec6992b0474706b8b3766518b0c274 with delete_type TABLET_DATA_DELETED (replaced by tablet a58c1c4aba5d4d5c9798e0ce0ee12839) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.782363 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet ac0813ac39364a268dd7df98d32403c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:31.782797 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 328bb280f36d4c79b2795c10df2b836b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:31.783586 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ad9baf131a1948d8ae5fae4c706aa726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad9baf131a1948d8ae5fae4c706aa726 already in progress: creating tablet
W20240503 14:12:31.783860 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ceec6992b0474706b8b3766518b0c274 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceec6992b0474706b8b3766518b0c274 already in progress: creating tablet
W20240503 14:12:31.784054 28017 catalog_manager.cc:4797] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet ac0813ac39364a268dd7df98d32403c6 because the tablet was not found. No further retry: Not found: Tablet not found: ac0813ac39364a268dd7df98d32403c6
I20240503 14:12:31.783993 28185 tablet_service.cc:1467] Processing CreateTablet for tablet a5a62a4abd9d4c8dbf642364faad8539 (DEFAULT_TABLE table=default.Table_196b6e1c710b483889aef9e34ab1d8fd [id=1b8310a6249a4a5e8251713859e0a4da]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:31.784284 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 328bb280f36d4c79b2795c10df2b836b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 328bb280f36d4c79b2795c10df2b836b already in progress: creating tablet
I20240503 14:12:31.784476 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a62a4abd9d4c8dbf642364faad8539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.775517 30666 raft_consensus.cc:482] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.784793 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9baf131a1948d8ae5fae4c706aa726 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.784757 30666 raft_consensus.cc:504] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.785164 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet ceec6992b0474706b8b3766518b0c274 with delete_type TABLET_DATA_DELETED (replaced by tablet a58c1c4aba5d4d5c9798e0ce0ee12839) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.785490 30666 leader_election.cc:288] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.785495 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet ac0813ac39364a268dd7df98d32403c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.785652 28661 rpcz_store.cc:276] 0503 14:12:30.317939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.318278 (+ 339us) service_pool.cc:225] Handling call
0503 14:12:31.599161 (+1280883us) ts_tablet_manager.cc:1120] Deleting tablet a035a5249f8348b591b27ba3fa7ae43a
0503 14:12:31.599176 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.651515 (+ 52339us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.732816 (+ 81301us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.774910 (+ 42094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280751}
I20240503 14:12:31.785862 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 328bb280f36d4c79b2795c10df2b836b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:31.786664 28459 tablet_service.cc:1467] Processing CreateTablet for tablet a5a62a4abd9d4c8dbf642364faad8539 (DEFAULT_TABLE table=default.Table_196b6e1c710b483889aef9e34ab1d8fd [id=1b8310a6249a4a5e8251713859e0a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.787504 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 682fe6ff2b4844019c645b9b877d9415 (DEFAULT_TABLE table=default.Table_c4e6f02aa70d498cabad3811f2fea24a [id=ab5d35f5d4d944799f5e5e51dcaab5f4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:31.788990 28427 leader_election.cc:341] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: aac51b83abe14ab5b1fb5d627233e58e
I20240503 14:12:31.789021 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 682fe6ff2b4844019c645b9b877d9415 (DEFAULT_TABLE table=default.Table_c4e6f02aa70d498cabad3811f2fea24a [id=ab5d35f5d4d944799f5e5e51dcaab5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.789425 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682fe6ff2b4844019c645b9b877d9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.790230 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6e7390ee45af41cb942b080a7f37b139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:31.790863 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e7390ee45af41cb942b080a7f37b139 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:31.790941 28600 raft_consensus.cc:348] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.791128 28600 raft_consensus.cc:374] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.791172 28600 raft_consensus.cc:729] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.791396 28600 consensus_queue.cc:260] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:31.791901 28427 leader_election.cc:341] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.791972 28600 ts_tablet_manager.cc:1398] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20240503 14:12:31.792295 28600 tablet_bootstrap.cc:492] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.798748 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a62a4abd9d4c8dbf642364faad8539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.803601 28600 tablet_bootstrap.cc:654] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:31.803645 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682fe6ff2b4844019c645b9b877d9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.788305 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fe462c9ee54a27b9714f6457036d76" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:31.791621 28345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac51b83abe14ab5b1fb5d627233e58e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.792236 30646 tablet.cc:1722] T 7261088abbc641db8d737a48bbac25bd P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:12:31.782371 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e220336186f648419b186feada6f0124" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:31.806038 28661 tablet_replica.cc:330] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:31.806267 28661 raft_consensus.cc:2229] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:31.806720 28661 raft_consensus.cc:2258] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:31.808205 28427 consensus_peers.cc:485] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34238c9859724d97a32f2a1d9315edb1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:31.780340 29828 ts_tablet_manager.cc:1888] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:31.814255 29828 log.cc:1196] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/0c7779f7a9544e17aea7135eab2ccd5f
I20240503 14:12:31.826740 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 007a892ef8504b848a400c05740eaa0c (table default.Table_a2bb32853fa94122b1e4188e3fc4f65e [id=66677901cea2405cbfa3403b540259fa]) successfully deleted
W20240503 14:12:31.827280 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 02d45788ae644a8581acca54cc55dd3f because the tablet was not found. No further retry: Not found: Tablet not found: 02d45788ae644a8581acca54cc55dd3f
I20240503 14:12:31.827641 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6de9e159d2ba47039dd6bd207a9cddeb (table default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a [id=f41fc50b1ca142f290aabb60f6cf0cf8]) successfully deleted
I20240503 14:12:31.828874 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5499b6d620467da6efd5e2240b0ae8 (DEFAULT_TABLE table=default.Table_78d5622f17704a729d8fe709d43e07c0 [id=09f51bfc0f0f44429ce8fdd4005f48b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.829455 30638 raft_consensus.cc:482] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.829617 28457 tablet_service.cc:1467] Processing CreateTablet for tablet f2915b66e6fd439fad3dbf09f8252ab8 (DEFAULT_TABLE table=default.Table_79162d4c00234c2eacb31954696e0e74 [id=abfd7b702ac9429ca4b540cdca507d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.829620 30638 raft_consensus.cc:504] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.830085 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2915b66e6fd439fad3dbf09f8252ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.830421 30638 leader_election.cc:288] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:31.832286 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c8a97076a44f9eaa3a15026f90a304" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:31.833181 28150 leader_election.cc:341] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.833297 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c8a97076a44f9eaa3a15026f90a304" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:31.833899 28150 leader_election.cc:341] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.834028 28150 leader_election.cc:302] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:31.834306 30638 raft_consensus.cc:2726] T d4c8a97076a44f9eaa3a15026f90a304 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.836499 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5499b6d620467da6efd5e2240b0ae8 (DEFAULT_TABLE table=default.Table_78d5622f17704a729d8fe709d43e07c0 [id=09f51bfc0f0f44429ce8fdd4005f48b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.837217 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5499b6d620467da6efd5e2240b0ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.837421 28661 ts_tablet_manager.cc:1875] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.838028 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 49445f0f1dd1451c9ba066e61273f2f1 (table default.Table_e4c32619dd494186a5a26bbc9cc04dbe [id=9c32d9f21eb94e69a164755faad557f2]) successfully deleted
I20240503 14:12:31.839277 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet b476e87faea1415d95ae014a93fa22af (table default.Table_6932067dd9e648e0a368cea1aae2eaa3 [id=99198f9902d14ce2ae33b8f229f36cf7]) successfully deleted
I20240503 14:12:31.841065 29828 ts_tablet_manager.cc:1909] T 0c7779f7a9544e17aea7135eab2ccd5f P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.841661 29828 tablet_replica.cc:330] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.841904 29828 raft_consensus.cc:2229] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.842110 29828 raft_consensus.cc:2258] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:31.841269 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2cad43f3c6094253ba87d0ed3fc29fd9 because the tablet was not found. No further retry: Not found: Tablet not found: 2cad43f3c6094253ba87d0ed3fc29fd9
I20240503 14:12:31.844017 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 11e14e8b5f4046029af4673a277112d9 (table default.Table_68199a1d18244f54bb0b7b0256a58e2a [id=2528ba5e3fe74c40949dcc7c289d0630]) successfully deleted
I20240503 14:12:31.845883 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5499b6d620467da6efd5e2240b0ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.850468 29828 ts_tablet_manager.cc:1875] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:31.850569 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 65e23e136ac94f9fa4180e15a81122a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e23e136ac94f9fa4180e15a81122a5 already in progress: creating tablet
I20240503 14:12:31.852385 28600 tablet_bootstrap.cc:492] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.852581 28600 ts_tablet_manager.cc:1367] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.009s
I20240503 14:12:31.855062 28600 raft_consensus.cc:348] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.855315 28600 raft_consensus.cc:374] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.855373 28600 raft_consensus.cc:729] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.855598 28600 consensus_queue.cc:260] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:31.856541 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 471992c1f7e84ef1953cd85448a492ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 471992c1f7e84ef1953cd85448a492ec already in progress: creating tablet
I20240503 14:12:31.857885 28600 ts_tablet_manager.cc:1398] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20240503 14:12:31.861161 28427 leader_election.cc:341] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.864537 28427 leader_election.cc:341] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.865842 28427 leader_election.cc:341] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.866438 30558 consensus_queue.cc:1035] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:31.868475 28600 tablet_bootstrap.cc:492] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.871784 28600 tablet_bootstrap.cc:654] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:31.872695 28427 leader_election.cc:341] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.872870 28427 leader_election.cc:302] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:31.873242 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:31.873610 30446 raft_consensus.cc:2726] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.873979 30616 raft_consensus.cc:482] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.874217 30616 raft_consensus.cc:504] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.874280 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f1ccc26f50432c9ef8789c01ef3519" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:31.874981 30616 leader_election.cc:288] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.876549 28600 tablet_bootstrap.cc:492] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:31.876749 28600 ts_tablet_manager.cc:1367] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:12:31.877787 28600 raft_consensus.cc:348] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.878085 28600 raft_consensus.cc:374] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.878166 28600 raft_consensus.cc:729] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.878422 28600 consensus_queue.cc:260] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:31.879973 28427 consensus_peers.cc:485] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:31.880431 30446 consensus_queue.cc:1035] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:31.882269 28427 leader_election.cc:341] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.881194 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9cad0a4cdfa8471bbd7066b91e85022e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cad0a4cdfa8471bbd7066b91e85022e already in progress: creating tablet
I20240503 14:12:31.883905 28600 ts_tablet_manager.cc:1398] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:31.884320 28600 tablet_bootstrap.cc:492] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:31.886431 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 34238c9859724d97a32f2a1d9315edb1 (table default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]) successfully deleted
W20240503 14:12:31.886760 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6fbf20a347f54724a0ce807436863785 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fbf20a347f54724a0ce807436863785 already in progress: creating tablet
W20240503 14:12:31.886973 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet aad397ed76ea4cf8820312ba1362512f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aad397ed76ea4cf8820312ba1362512f already in progress: creating tablet
I20240503 14:12:31.887135 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet a035a5249f8348b591b27ba3fa7ae43a (table default.Table_848d74c659274863a2c1046f6de2568e [id=ed5de25a209e4d3f944021a08a498d0e]) successfully deleted
I20240503 14:12:31.888424 28179 tablet_service.cc:1467] Processing CreateTablet for tablet f2915b66e6fd439fad3dbf09f8252ab8 (DEFAULT_TABLE table=default.Table_79162d4c00234c2eacb31954696e0e74 [id=abfd7b702ac9429ca4b540cdca507d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.895239 28600 tablet_bootstrap.cc:654] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:31.898536 28290 leader_election.cc:341] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 646f83185d5241268d4126d10561e937
I20240503 14:12:31.899238 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebbb349fb274b0dbffeacbd89b26b36 (DEFAULT_TABLE table=default.Table_bfa7ab95de17487db85d9c7e73e24e4b [id=ce3080473998421aa0466b685dd71980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.899789 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 01482b478dcd45598fa65d14904fe276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.900204 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 61da48859c4d4a73bb1b9d7c0ede8106 with delete_type TABLET_DATA_DELETED (replaced by tablet 27b25518570e4f2eb2666d420a9ef39c) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.900696 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet a88cf5de223f4a6abc21691fea1c4da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:30 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.901103 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 530f24322a454f7bb04e818ca865a369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.901464 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet f783ee62adba49579a0518b240b795bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.901944 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 9028a6990d0840ffa6849143f6c004db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.902313 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 989ea2f962a7492d8c32e4becec1457d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.902930 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 201cbe667ff640cf93e24f4bfcf113e3 (DEFAULT_TABLE table=default.Table_1f1afeacc77b4bf4af1a39ce67f957b3 [id=8dd887797f0f4d2db696c41352a828c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.903779 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9baf131a1948d8ae5fae4c706aa726 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.904794 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebbb349fb274b0dbffeacbd89b26b36 (DEFAULT_TABLE table=default.Table_bfa7ab95de17487db85d9c7e73e24e4b [id=ce3080473998421aa0466b685dd71980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.905293 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70456b08214699869453212fb683ef (DEFAULT_TABLE table=default.Table_1e60a3851c8f4b49985e97e1d6096ac0 [id=971926703202468d96dd9af5449c3100]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.905831 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebbb349fb274b0dbffeacbd89b26b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.908411 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70456b08214699869453212fb683ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.909255 28614 tablet_bootstrap.cc:492] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:31.909452 28614 ts_tablet_manager.cc:1367] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.198s user 0.009s sys 0.007s
I20240503 14:12:31.910220 28614 raft_consensus.cc:348] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.910454 28614 raft_consensus.cc:374] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.910506 28614 raft_consensus.cc:729] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.910725 28614 consensus_queue.cc:260] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.911301 28614 ts_tablet_manager.cc:1398] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:31.911652 28614 tablet_bootstrap.cc:492] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:31.918233 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2915b66e6fd439fad3dbf09f8252ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.918500 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebbb349fb274b0dbffeacbd89b26b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.926548 29828 ts_tablet_manager.cc:1888] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:31.929919 29828 log.cc:1196] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a035a5249f8348b591b27ba3fa7ae43a
I20240503 14:12:31.930526 29828 ts_tablet_manager.cc:1909] T a035a5249f8348b591b27ba3fa7ae43a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:31.931279 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1142) took 1002 ms. Trace:
I20240503 14:12:31.931548 29828 rpcz_store.cc:276] 0503 14:12:30.929182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.933797 (+ 4615us) service_pool.cc:225] Handling call
0503 14:12:31.841605 (+907808us) ts_tablet_manager.cc:1120] Deleting tablet a035a5249f8348b591b27ba3fa7ae43a
0503 14:12:31.841613 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.869569 (+ 27956us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.926417 (+ 56848us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:31.931220 (+ 4803us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":907559}
I20240503 14:12:31.932924 29828 tablet_replica.cc:330] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:31.933981 29828 raft_consensus.cc:2229] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.938215 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70456b08214699869453212fb683ef (DEFAULT_TABLE table=default.Table_1e60a3851c8f4b49985e97e1d6096ac0 [id=971926703202468d96dd9af5449c3100]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.937948 29828 raft_consensus.cc:2258] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.939064 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70456b08214699869453212fb683ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.943217 29828 ts_tablet_manager.cc:1875] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:31.945111 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 8113a36f49ff4b29b32a0532d1aabebd (DEFAULT_TABLE table=default.Table_435f8c6696444d43a33900aee0682c9a [id=dfc716c4d90e411e963b25a32772debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.947929 28466 tablet_service.cc:1467] Processing CreateTablet for tablet 8113a36f49ff4b29b32a0532d1aabebd (DEFAULT_TABLE table=default.Table_435f8c6696444d43a33900aee0682c9a [id=dfc716c4d90e411e963b25a32772debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.948498 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8113a36f49ff4b29b32a0532d1aabebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.949323 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet ceec6992b0474706b8b3766518b0c274 with delete_type TABLET_DATA_DELETED (replaced by tablet a58c1c4aba5d4d5c9798e0ce0ee12839) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.949838 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet ac0813ac39364a268dd7df98d32403c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.950529 28318 tablet_service.cc:1467] Processing CreateTablet for tablet a5a62a4abd9d4c8dbf642364faad8539 (DEFAULT_TABLE table=default.Table_196b6e1c710b483889aef9e34ab1d8fd [id=1b8310a6249a4a5e8251713859e0a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.951323 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 328bb280f36d4c79b2795c10df2b836b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:31.952018 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 682fe6ff2b4844019c645b9b877d9415 (DEFAULT_TABLE table=default.Table_c4e6f02aa70d498cabad3811f2fea24a [id=ab5d35f5d4d944799f5e5e51dcaab5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.953145 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5499b6d620467da6efd5e2240b0ae8 (DEFAULT_TABLE table=default.Table_78d5622f17704a729d8fe709d43e07c0 [id=09f51bfc0f0f44429ce8fdd4005f48b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.953482 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebbb349fb274b0dbffeacbd89b26b36 (DEFAULT_TABLE table=default.Table_bfa7ab95de17487db85d9c7e73e24e4b [id=ce3080473998421aa0466b685dd71980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.953874 28329 tablet_service.cc:1467] Processing CreateTablet for tablet f2915b66e6fd439fad3dbf09f8252ab8 (DEFAULT_TABLE table=default.Table_79162d4c00234c2eacb31954696e0e74 [id=abfd7b702ac9429ca4b540cdca507d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.954459 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70456b08214699869453212fb683ef (DEFAULT_TABLE table=default.Table_1e60a3851c8f4b49985e97e1d6096ac0 [id=971926703202468d96dd9af5449c3100]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.954993 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 8113a36f49ff4b29b32a0532d1aabebd (DEFAULT_TABLE table=default.Table_435f8c6696444d43a33900aee0682c9a [id=dfc716c4d90e411e963b25a32772debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.956354 28216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5189b6cdaa486c8c2f1b5c141d513b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:31.958158 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5189b6cdaa486c8c2f1b5c141d513b" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:31.960494 28427 leader_election.cc:341] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:31.961038 28427 leader_election.cc:341] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.961228 28427 leader_election.cc:302] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:31.961556 30446 raft_consensus.cc:2726] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:31.961997 28427 leader_election.cc:341] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:31.962177 28427 leader_election.cc:302] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:31.962841 30446 raft_consensus.cc:2726] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:31.969868 28186 tablet_service.cc:1467] Processing CreateTablet for tablet ac0813ac39364a268dd7df98d32403c6 (DEFAULT_TABLE table=default.Table_38b33f45a641422ba567e4bbadaa6e23 [id=62f9133076454b1092f000a128689e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.973127 30616 raft_consensus.cc:482] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:31.973212 28604 tablet_bootstrap.cc:492] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:31.973327 30616 raft_consensus.cc:504] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:31.973567 28604 ts_tablet_manager.cc:1367] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.209s user 0.030s sys 0.031s
I20240503 14:12:31.974026 30616 leader_election.cc:288] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:31.976562 28604 raft_consensus.cc:348] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.977314 28187 tablet_service.cc:1467] Processing CreateTablet for tablet fa3120e8569c4fdba324eb0b73b35675 (DEFAULT_TABLE table=default.Table_c25685742fa44084940ddfc0a0b5cb9a [id=54f4bacc8d464e4d8b4db84f2c26fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.977456 28604 raft_consensus.cc:374] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:31.977864 28604 raft_consensus.cc:729] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.978317 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9f6a750b744433b01f821506767a02 (DEFAULT_TABLE table=default.Table_99b171729b6f4232a36bb9ba33008749 [id=39dbc007b7a5429a98e1cfa1af884c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.978479 28466 tablet_service.cc:1467] Processing CreateTablet for tablet ac0813ac39364a268dd7df98d32403c6 (DEFAULT_TABLE table=default.Table_38b33f45a641422ba567e4bbadaa6e23 [id=62f9133076454b1092f000a128689e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.978940 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0813ac39364a268dd7df98d32403c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.979048 28604 consensus_queue.cc:260] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:31.985569 28604 ts_tablet_manager.cc:1398] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.012s user 0.003s sys 0.001s
I20240503 14:12:31.987673 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8113a36f49ff4b29b32a0532d1aabebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.993841 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201cbe667ff640cf93e24f4bfcf113e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.994246 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebbb349fb274b0dbffeacbd89b26b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.994480 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2915b66e6fd439fad3dbf09f8252ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.994666 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8113a36f49ff4b29b32a0532d1aabebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.996233 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70456b08214699869453212fb683ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.996513 28459 tablet_service.cc:1467] Processing CreateTablet for tablet fa3120e8569c4fdba324eb0b73b35675 (DEFAULT_TABLE table=default.Table_c25685742fa44084940ddfc0a0b5cb9a [id=54f4bacc8d464e4d8b4db84f2c26fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:31.996958 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3120e8569c4fdba324eb0b73b35675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.000228 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a62a4abd9d4c8dbf642364faad8539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.000411 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0813ac39364a268dd7df98d32403c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.000789 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3120e8569c4fdba324eb0b73b35675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.003348 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9f6a750b744433b01f821506767a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.003952 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682fe6ff2b4844019c645b9b877d9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.007421 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 6e910e098ec44a07af6ed8ff4647fc7e (DEFAULT_TABLE table=default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.007658 28614 tablet_bootstrap.cc:654] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.007843 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e910e098ec44a07af6ed8ff4647fc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.008920 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5499b6d620467da6efd5e2240b0ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.012228 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 03621419374d455c96ecc4d4813a6b03 (DEFAULT_TABLE table=default.Table_ddf2834af03d458b917b8fcbd0852de3 [id=0bd6451c3a94499ba33aa1e981b68f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.012696 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03621419374d455c96ecc4d4813a6b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.014948 28661 ts_tablet_manager.cc:1888] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:32.017218 28604 tablet_bootstrap.cc:492] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:32.019680 28604 tablet_bootstrap.cc:654] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.028285 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 3722e89584594fd09ac41a0d79add90d (DEFAULT_TABLE table=default.Table_cb05adc8775b48d1837187d135ce1201 [id=c728632b8c404533afc207c5026e4359]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.028285 28461 tablet_service.cc:1467] Processing CreateTablet for tablet 03621419374d455c96ecc4d4813a6b03 (DEFAULT_TABLE table=default.Table_ddf2834af03d458b917b8fcbd0852de3 [id=0bd6451c3a94499ba33aa1e981b68f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.029114 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9f6a750b744433b01f821506767a02 (DEFAULT_TABLE table=default.Table_99b171729b6f4232a36bb9ba33008749 [id=39dbc007b7a5429a98e1cfa1af884c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.029438 28661 log.cc:1196] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/6de9e159d2ba47039dd6bd207a9cddeb
I20240503 14:12:32.029484 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 6e910e098ec44a07af6ed8ff4647fc7e (DEFAULT_TABLE table=default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.030344 30616 raft_consensus.cc:482] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.030503 30616 raft_consensus.cc:504] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.030983 29828 ts_tablet_manager.cc:1888] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:32.031111 29828 log.cc:1196] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/3dfd6d40a57e4bea973874ff12c0ea3a
I20240503 14:12:32.031251 30616 leader_election.cc:288] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.031450 29828 ts_tablet_manager.cc:1909] T 3dfd6d40a57e4bea973874ff12c0ea3a P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:32.032428 28661 ts_tablet_manager.cc:1909] T 6de9e159d2ba47039dd6bd207a9cddeb P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:32.033601 30446 raft_consensus.cc:482] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.033771 30446 raft_consensus.cc:504] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.034467 30446 leader_election.cc:288] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.034946 28604 tablet_bootstrap.cc:492] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:32.035108 28604 ts_tablet_manager.cc:1367] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
W20240503 14:12:32.035982 28427 leader_election.cc:341] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 9b21cd1262d94d1c994b29c44f86cb5e
I20240503 14:12:32.036049 28604 raft_consensus.cc:348] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.036322 28604 raft_consensus.cc:374] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.036382 28604 raft_consensus.cc:729] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.036609 28604 consensus_queue.cc:260] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.037129 28604 ts_tablet_manager.cc:1398] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:32.042824 30638 raft_consensus.cc:482] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.043002 30638 raft_consensus.cc:504] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.043697 30638 leader_election.cc:288] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.044617 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e330b28d642a47e69e427b56a90d9807" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:32.045361 28150 leader_election.cc:341] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.043529 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 0c7779f7a9544e17aea7135eab2ccd5f (table default.Table_2a170feff0884500b519684ac43a06b0 [id=2d1a26ea051048589afec842f2a31c72]) successfully deleted
I20240503 14:12:32.047055 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3722e89584594fd09ac41a0d79add90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.048538 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03621419374d455c96ecc4d4813a6b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.052336 30446 raft_consensus.cc:482] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.052498 30446 raft_consensus.cc:504] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.053192 30446 leader_election.cc:288] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.053817 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:32.054052 28208 raft_consensus.cc:2454] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:32.055548 28427 leader_election.cc:302] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:32.056898 30446 raft_consensus.cc:2781] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:32.056994 30446 raft_consensus.cc:482] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:32.057049 30446 raft_consensus.cc:3037] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.044797 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e330b28d642a47e69e427b56a90d9807" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.057765 28150 leader_election.cc:341] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.057904 28150 leader_election.cc:302] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.058064 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1144) took 1128 ms. Trace:
I20240503 14:12:32.058159 29828 rpcz_store.cc:276] 0503 14:12:30.929414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.934132 (+ 4718us) service_pool.cc:225] Handling call
0503 14:12:31.932642 (+998510us) ts_tablet_manager.cc:1120] Deleting tablet 3dfd6d40a57e4bea973874ff12c0ea3a
0503 14:12:31.932656 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.022403 (+ 89747us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.030851 (+ 8448us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.058041 (+ 27190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998217}
I20240503 14:12:32.058348 30638 raft_consensus.cc:2726] T e330b28d642a47e69e427b56a90d9807 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.058490 29828 tablet_replica.cc:330] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:32.058677 29828 raft_consensus.cc:2229] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.058830 29828 pending_rounds.cc:62] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:32.058864 29828 pending_rounds.cc:68] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597769678561280 op_type: NO_OP noop_request { }
I20240503 14:12:32.058971 29828 raft_consensus.cc:2866] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:32.059044 29828 raft_consensus.cc:2258] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.061695 28183 tablet_service.cc:1467] Processing CreateTablet for tablet 3722e89584594fd09ac41a0d79add90d (DEFAULT_TABLE table=default.Table_cb05adc8775b48d1837187d135ce1201 [id=c728632b8c404533afc207c5026e4359]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.059952 30446 raft_consensus.cc:504] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.071342 30446 leader_election.cc:288] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.071864 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9f6a750b744433b01f821506767a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.074891 29828 ts_tablet_manager.cc:1875] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:12:32.077739 28427 consensus_peers.cc:485] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 989ea2f962a7492d8c32e4becec1457d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:32.079182 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e910e098ec44a07af6ed8ff4647fc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.082177 28604 tablet_bootstrap.cc:492] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:32.082712 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1133) took 1765 ms. Trace:
I20240503 14:12:32.082901 28661 rpcz_store.cc:276] 0503 14:12:30.317503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.318496 (+ 993us) service_pool.cc:225] Handling call
0503 14:12:31.788548 (+1470052us) ts_tablet_manager.cc:1120] Deleting tablet 6de9e159d2ba47039dd6bd207a9cddeb
0503 14:12:31.805956 (+ 17408us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:31.893408 (+ 87452us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.014810 (+121402us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.082683 (+ 67873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469859}
I20240503 14:12:32.083565 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1141) took 1433 ms. Trace:
I20240503 14:12:32.083724 28661 rpcz_store.cc:276] 0503 14:12:30.649838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.652712 (+ 2874us) service_pool.cc:225] Handling call
0503 14:12:32.083396 (+1430684us) ts_tablet_manager.cc:1120] Deleting tablet 65e23e136ac94f9fa4180e15a81122a5
0503 14:12:32.083409 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.083540 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419026}
I20240503 14:12:32.084233 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1146) took 1430 ms. Trace:
I20240503 14:12:32.084407 28604 tablet_bootstrap.cc:654] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.077374 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:32.084374 28661 rpcz_store.cc:276] 0503 14:12:30.668690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.680183 (+ 11493us) service_pool.cc:225] Handling call
0503 14:12:32.084108 (+1403925us) ts_tablet_manager.cc:1120] Deleting tablet 6fbf20a347f54724a0ce807436863785
0503 14:12:32.084119 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.084215 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403675}
I20240503 14:12:32.088835 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b21cd1262d94d1c994b29c44f86cb5e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.091683 28661 tablet_replica.cc:330] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:32.092330 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.092669 28661 raft_consensus.cc:2229] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:32.093276 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
W20240503 14:12:32.094890 28290 leader_election.cc:341] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 646f83185d5241268d4126d10561e937
I20240503 14:12:32.095034 28290 leader_election.cc:302] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.095279 30616 raft_consensus.cc:2726] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.097913 28290 leader_election.cc:302] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.098345 30616 raft_consensus.cc:2726] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.101997 29828 ts_tablet_manager.cc:1888] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:32.102166 29828 log.cc:1196] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/cf810e84367f420b928a171790849445
I20240503 14:12:32.102592 29828 ts_tablet_manager.cc:1909] T cf810e84367f420b928a171790849445 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:32.102973 29828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58028 (request call id 1145) took 1173 ms. Trace:
I20240503 14:12:32.103070 29828 rpcz_store.cc:276] 0503 14:12:30.929569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.934345 (+ 4776us) service_pool.cc:225] Handling call
0503 14:12:32.058435 (+1124090us) ts_tablet_manager.cc:1120] Deleting tablet cf810e84367f420b928a171790849445
0503 14:12:32.058446 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.098586 (+ 40140us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.101860 (+ 3274us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.102949 (+ 1089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123900}
I20240503 14:12:32.108203 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a035a5249f8348b591b27ba3fa7ae43a (table default.Table_848d74c659274863a2c1046f6de2568e [id=ed5de25a209e4d3f944021a08a498d0e]) successfully deleted
I20240503 14:12:32.120792 28208 raft_consensus.cc:3037] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.123945 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9f6a750b744433b01f821506767a02 (DEFAULT_TABLE table=default.Table_99b171729b6f4232a36bb9ba33008749 [id=39dbc007b7a5429a98e1cfa1af884c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.124125 28208 raft_consensus.cc:2454] T 888636e9beca43edab96a8736751409e P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:32.124558 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9f6a750b744433b01f821506767a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.125241 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3722e89584594fd09ac41a0d79add90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.124806 28335 tablet_service.cc:1467] Processing CreateTablet for tablet fa3120e8569c4fdba324eb0b73b35675 (DEFAULT_TABLE table=default.Table_c25685742fa44084940ddfc0a0b5cb9a [id=54f4bacc8d464e4d8b4db84f2c26fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.128594 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3120e8569c4fdba324eb0b73b35675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.130421 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 6e910e098ec44a07af6ed8ff4647fc7e (DEFAULT_TABLE table=default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.130872 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e910e098ec44a07af6ed8ff4647fc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.124559 28329 tablet_service.cc:1467] Processing CreateTablet for tablet ac0813ac39364a268dd7df98d32403c6 (DEFAULT_TABLE table=default.Table_38b33f45a641422ba567e4bbadaa6e23 [id=62f9133076454b1092f000a128689e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.136372 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0813ac39364a268dd7df98d32403c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.137504 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 03621419374d455c96ecc4d4813a6b03 (DEFAULT_TABLE table=default.Table_ddf2834af03d458b917b8fcbd0852de3 [id=0bd6451c3a94499ba33aa1e981b68f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.137925 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03621419374d455c96ecc4d4813a6b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.138490 30638 raft_consensus.cc:482] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.138665 30638 raft_consensus.cc:504] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.139487 30638 leader_election.cc:288] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.139959 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69e01bdbae349d0a61a295b6f6352f0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.140550 28150 leader_election.cc:341] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.141141 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 3722e89584594fd09ac41a0d79add90d (DEFAULT_TABLE table=default.Table_cb05adc8775b48d1837187d135ce1201 [id=c728632b8c404533afc207c5026e4359]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.141541 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3722e89584594fd09ac41a0d79add90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.143828 28600 tablet_bootstrap.cc:492] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.144001 28600 ts_tablet_manager.cc:1367] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.260s user 0.022s sys 0.029s
I20240503 14:12:32.144791 28600 raft_consensus.cc:348] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.145001 28600 raft_consensus.cc:374] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.145048 28600 raft_consensus.cc:729] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.145242 28600 consensus_queue.cc:260] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.149852 28600 ts_tablet_manager.cc:1398] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:32.150295 28600 tablet_bootstrap.cc:492] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.152518 28600 tablet_bootstrap.cc:654] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.155894 30646 tablet.cc:1717] T d832cd30f2414303818d34a0cdf47e60 P d3a6b9b7913444d29c96581af6febf33: Skipping requested alter to schema version 3, tablet already version 3
I20240503 14:12:32.156723 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_7757fd0ba3ea4c5c85b42c1e3e3facc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.157362 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7757fd0ba3ea4c5c85b42c1e3e3facc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.158390 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5e84413c67034dc3ac76040484753c93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.158993 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e84413c67034dc3ac76040484753c93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:32.159669 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d70456b08214699869453212fb683ef consensus state has no leader
I20240503 14:12:32.160054 28033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d0f835c02d5f4ffea64369bb8db62de9" } new_table_name: "default.Table_8f5381838b204ba5b4c29ce00f05044a" modify_external_catalogs: true
I20240503 14:12:32.160320 28290 leader_election.cc:302] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.160634 30616 raft_consensus.cc:2726] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.161813 28614 tablet_bootstrap.cc:492] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.161870 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_4d04603a0852462f8671107845e91561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:32.162042 28614 ts_tablet_manager.cc:1367] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.250s user 0.011s sys 0.007s
W20240503 14:12:32.162276 28290 leader_election.cc:341] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: a31555fc800142bd849dbd572a194002
W20240503 14:12:32.162442 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d04603a0852462f8671107845e91561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240503 14:12:32.162846 28290 leader_election.cc:341] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 6696a237c3dd428097e2ebdf24afd63f
I20240503 14:12:32.163275 28614 raft_consensus.cc:348] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.163643 28614 raft_consensus.cc:374] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.163800 28614 raft_consensus.cc:729] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.164171 28614 consensus_queue.cc:260] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.164914 28614 ts_tablet_manager.cc:1398] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:32.173640 28614 tablet_bootstrap.cc:492] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.177186 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 6de9e159d2ba47039dd6bd207a9cddeb (table default.Table_a3b79a0d3c6d4b45b67ba67fbf38744a [id=f41fc50b1ca142f290aabb60f6cf0cf8]) successfully deleted
W20240503 14:12:32.116652 28427 leader_election.cc:341] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.178033 28427 leader_election.cc:302] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:32.178357 30558 raft_consensus.cc:2726] T 9b21cd1262d94d1c994b29c44f86cb5e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.140076 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69e01bdbae349d0a61a295b6f6352f0" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.117389 30446 raft_consensus.cc:482] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:12:32.179350 28150 leader_election.cc:341] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.179394 30446 raft_consensus.cc:504] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.179502 28150 leader_election.cc:302] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.179821 30638 raft_consensus.cc:2726] T e69e01bdbae349d0a61a295b6f6352f0 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.181231 28614 tablet_bootstrap.cc:654] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.181450 30446 leader_election.cc:288] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:32.182300 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 65664.1us
W20240503 14:12:32.183012 28427 leader_election.cc:341] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:32.183455 28427 leader_election.cc:341] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.172444 28345 raft_consensus.cc:3037] T 86f1ccc26f50432c9ef8789c01ef3519 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:32.170778 28290 leader_election.cc:341] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: 6696a237c3dd428097e2ebdf24afd63f
I20240503 14:12:32.186722 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167a926fa61747e2b4fcb52ba4eeea5b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:32.186882 29828 tablet_replica.cc:330] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:32.187139 28208 raft_consensus.cc:2454] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:32.189779 28427 leader_election.cc:302] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:32.190199 30558 raft_consensus.cc:2781] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:32.191025 30558 raft_consensus.cc:686] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:32.191402 30558 consensus_queue.cc:237] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.194785 28290 leader_election.cc:302] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:32.195367 28290 leader_election.cc:341] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Not found: Tablet not found: a31555fc800142bd849dbd572a194002
I20240503 14:12:32.195518 28290 leader_election.cc:302] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.195853 30616 raft_consensus.cc:2726] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.196497 30616 raft_consensus.cc:2726] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.197223 29828 raft_consensus.cc:2229] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.200008 28427 leader_election.cc:302] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:32.200461 30446 raft_consensus.cc:2781] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:32.200629 30446 raft_consensus.cc:482] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:32.200779 30446 raft_consensus.cc:3037] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.094470 28661 raft_consensus.cc:2258] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.202111 29828 raft_consensus.cc:2258] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.202991 29828 ts_tablet_manager.cc:1875] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.209674 28661 ts_tablet_manager.cc:1875] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.221496 30446 raft_consensus.cc:504] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.222484 30446 leader_election.cc:288] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
W20240503 14:12:32.229526 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 65e23e136ac94f9fa4180e15a81122a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e23e136ac94f9fa4180e15a81122a5 already in progress: creating tablet
W20240503 14:12:32.229988 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 6fbf20a347f54724a0ce807436863785 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fbf20a347f54724a0ce807436863785 already in progress: creating tablet
I20240503 14:12:32.230976 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 3dfd6d40a57e4bea973874ff12c0ea3a (table default.Table_4eec071da1eb4980b74b08ce5b21765b [id=50c1a152084e4b9690986b223b55e70a]) successfully deleted
I20240503 14:12:32.231413 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet cf810e84367f420b928a171790849445 (table default.Table_4515f7561eae476abe1e2f9b2cf55b1a [id=0c370c08243c4127a1e38025866ece8d]) successfully deleted
W20240503 14:12:32.231873 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 01482b478dcd45598fa65d14904fe276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01482b478dcd45598fa65d14904fe276 already in progress: creating tablet
I20240503 14:12:32.233485 29828 ts_tablet_manager.cc:1888] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:32.236028 30616 raft_consensus.cc:482] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.245980 30616 raft_consensus.cc:504] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:32.246805 30616 leader_election.cc:288] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:32.249420 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167a926fa61747e2b4fcb52ba4eeea5b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:32.249827 28208 raft_consensus.cc:3037] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:32.241091 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 45691.6us
I20240503 14:12:32.253971 30616 raft_consensus.cc:482] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.254169 30616 raft_consensus.cc:504] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.254896 30616 leader_election.cc:288] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.255509 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c79ffee3954b5c870f4a8aa6379517" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.255733 28345 raft_consensus.cc:3037] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.255746 28489 raft_consensus.cc:2454] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:32.257025 29828 log.cc:1196] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/989ea2f962a7492d8c32e4becec1457d
I20240503 14:12:32.253050 28208 raft_consensus.cc:2454] T 167a926fa61747e2b4fcb52ba4eeea5b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:32.260479 28343 raft_consensus.cc:3037] T ac5189b6cdaa486c8c2f1b5c141d513b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.268544 28614 tablet_bootstrap.cc:492] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.269197 28614 ts_tablet_manager.cc:1367] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.096s user 0.001s sys 0.011s
I20240503 14:12:32.270258 28614 raft_consensus.cc:348] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.275096 28614 raft_consensus.cc:374] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.275434 28614 raft_consensus.cc:729] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.274389 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167a926fa61747e2b4fcb52ba4eeea5b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.277201 28354 raft_consensus.cc:2454] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:32.271620 29828 ts_tablet_manager.cc:1909] T 989ea2f962a7492d8c32e4becec1457d P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:32.280284 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167a926fa61747e2b4fcb52ba4eeea5b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:32.280546 28356 raft_consensus.cc:3037] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.283375 28356 raft_consensus.cc:2454] T 167a926fa61747e2b4fcb52ba4eeea5b P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:32.283668 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdc868ddf0a490292eae78ebc8c8ec6" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.279285 28614 consensus_queue.cc:260] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.286161 28614 ts_tablet_manager.cc:1398] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 14:12:32.292520 28614 tablet_bootstrap.cc:492] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.294481 28614 tablet_bootstrap.cc:654] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:32.233098 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 61da48859c4d4a73bb1b9d7c0ede8106 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61da48859c4d4a73bb1b9d7c0ede8106 already in progress: creating tablet
I20240503 14:12:32.296852 28604 tablet_bootstrap.cc:492] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:32.297026 28604 ts_tablet_manager.cc:1367] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.215s user 0.020s sys 0.041s
I20240503 14:12:32.298151 28600 tablet_bootstrap.cc:492] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.298321 28600 ts_tablet_manager.cc:1367] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.148s user 0.014s sys 0.009s
I20240503 14:12:32.299027 28600 raft_consensus.cc:348] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.299244 28600 raft_consensus.cc:374] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.299299 28600 raft_consensus.cc:729] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.299515 28600 consensus_queue.cc:260] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.300030 28600 ts_tablet_manager.cc:1398] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:32.301004 30558 consensus_queue.cc:1035] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:32.301914 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 68813.6us
W20240503 14:12:32.302310 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a88cf5de223f4a6abc21691fea1c4da0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88cf5de223f4a6abc21691fea1c4da0 already in progress: creating tablet
W20240503 14:12:32.302573 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 530f24322a454f7bb04e818ca865a369 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 530f24322a454f7bb04e818ca865a369 already in progress: creating tablet
W20240503 14:12:32.302755 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f783ee62adba49579a0518b240b795bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f783ee62adba49579a0518b240b795bc already in progress: creating tablet
W20240503 14:12:32.302932 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 9028a6990d0840ffa6849143f6c004db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9028a6990d0840ffa6849143f6c004db already in progress: creating tablet
I20240503 14:12:32.303619 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 989ea2f962a7492d8c32e4becec1457d (table default.Table_da6c5739b62c43758e9e53182b790622 [id=6b342f8063bb4c2db3d8f63359afa1a8]) successfully deleted
I20240503 14:12:32.304399 28600 tablet_bootstrap.cc:492] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.305896 28600 tablet_bootstrap.cc:654] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.310962 28604 raft_consensus.cc:348] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.311822 28604 raft_consensus.cc:374] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.311985 28604 raft_consensus.cc:729] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.313961 28604 consensus_queue.cc:260] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.314734 28604 ts_tablet_manager.cc:1398] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 14:12:32.315117 28604 tablet_bootstrap.cc:492] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:32.323197 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ad9baf131a1948d8ae5fae4c706aa726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad9baf131a1948d8ae5fae4c706aa726 already in progress: creating tablet
W20240503 14:12:32.324734 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ceec6992b0474706b8b3766518b0c274 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceec6992b0474706b8b3766518b0c274 already in progress: creating tablet
W20240503 14:12:32.326694 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet ac0813ac39364a268dd7df98d32403c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac0813ac39364a268dd7df98d32403c6 already in progress: creating tablet
W20240503 14:12:32.328974 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 328bb280f36d4c79b2795c10df2b836b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 328bb280f36d4c79b2795c10df2b836b already in progress: creating tablet
I20240503 14:12:32.317301 28604 tablet_bootstrap.cc:654] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:32.320607 28427 consensus_peers.cc:485] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:32.332500 28427 leader_election.cc:302] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:32.332996 30558 raft_consensus.cc:2781] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:32.333212 30558 raft_consensus.cc:686] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:32.333542 30558 consensus_queue.cc:237] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.334312 28600 tablet_bootstrap.cc:492] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.334606 28600 ts_tablet_manager.cc:1367] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.001s
I20240503 14:12:32.332942 30666 consensus_queue.cc:1035] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:32.335525 28600 raft_consensus.cc:348] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.335913 28600 raft_consensus.cc:374] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.336030 28600 raft_consensus.cc:729] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.336355 28600 consensus_queue.cc:260] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.342041 28600 ts_tablet_manager.cc:1398] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:32.342554 28600 tablet_bootstrap.cc:492] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
W20240503 14:12:32.345474 28290 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.27.50.254:36571, user_credentials={real_user=slave}} blocked reactor thread for 48676.8us
W20240503 14:12:32.346148 28290 leader_election.cc:341] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: d7c79ffee3954b5c870f4a8aa6379517
W20240503 14:12:32.346570 28290 leader_election.cc:341] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 5cdc868ddf0a490292eae78ebc8c8ec6
I20240503 14:12:32.346940 28290 leader_election.cc:302] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:32.349388 28600 tablet_bootstrap.cc:654] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.349611 30616 raft_consensus.cc:2781] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:32.350143 30616 raft_consensus.cc:482] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:32.350886 30616 raft_consensus.cc:3037] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.353788 28661 ts_tablet_manager.cc:1888] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:32.354159 30616 raft_consensus.cc:504] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:32.355135 30616 leader_election.cc:288] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:32.357185 28290 leader_election.cc:302] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.357650 30616 raft_consensus.cc:2726] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.357941 28614 tablet_bootstrap.cc:492] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.358211 28614 ts_tablet_manager.cc:1367] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.008s
I20240503 14:12:32.359352 28614 raft_consensus.cc:348] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.360798 28614 raft_consensus.cc:374] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.361174 28614 raft_consensus.cc:729] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.361680 28614 consensus_queue.cc:260] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.363180 28614 ts_tablet_manager.cc:1398] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:32.364228 28614 tablet_bootstrap.cc:492] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.367650 28614 tablet_bootstrap.cc:654] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:32.370497 28290 leader_election.cc:341] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: d7c79ffee3954b5c870f4a8aa6379517
W20240503 14:12:32.379655 28033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 464f9dd19dfb4447b5faebc974671601 consensus state has no leader
I20240503 14:12:32.380363 28027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 7b86451bc448428da7485e70e0d54b89
I20240503 14:12:32.380829 28027 catalog_manager.cc:6554] ReplaceTablet: tablet 7b86451bc448428da7485e70e0d54b89 deleted and replaced by tablet 5cad03adb3ee419c962aaf20c72f8c04
I20240503 14:12:32.387569 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:32.394106 30593 log.cc:824] T 00000000000000000000000000000000 P bb6a7aee99214df4b02858ddae025b14: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:32.395902 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 7261088abbc641db8d737a48bbac25bd (table default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671]). Expected version 1 got 0
I20240503 14:12:32.396975 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:32.397864 28614 tablet_bootstrap.cc:492] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.398144 28614 ts_tablet_manager.cc:1367] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.001s
I20240503 14:12:32.398164 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 2f7b805aee554f8fa4c198ce3409c6e6 (table default.Table_ee9f2d2bd2b54111963a23deaf110216 [id=4df57b62a5b34ac182e799495a848041]). Expected version 1 got 0
I20240503 14:12:32.398878 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 788e4bfa1e414630939bc19da1ede3fb (table default.Table_b7b3cecd524e40feb9c6251d0403b764 [id=ddef27f860534921adab476e0f391782]). Expected version 2 got 0
I20240503 14:12:32.399185 28614 raft_consensus.cc:348] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.399514 28614 raft_consensus.cc:374] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.399614 28614 raft_consensus.cc:729] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.399953 28614 consensus_queue.cc:260] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.400640 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_ec97037ea5a849bfb48a72d6771fbf1f [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 2 got 0
I20240503 14:12:32.403301 28614 ts_tablet_manager.cc:1398] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:32.403770 28614 tablet_bootstrap.cc:492] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.404780 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_84e92c206ac54e07845732e6239ab59a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.405483 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84e92c206ac54e07845732e6239ab59a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.405936 28614 tablet_bootstrap.cc:654] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.409202 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7060c6b06e14403ad4ab81fc6e5c8cd" } modify_external_catalogs: true
I20240503 14:12:32.409446 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a7060c6b06e14403ad4ab81fc6e5c8cd" } modify_external_catalogs: true
I20240503 14:12:32.413190 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7534f990ae504fa5aa364d6948a1760c" } modify_external_catalogs: true
I20240503 14:12:32.413398 28403 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
I20240503 14:12:32.413456 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_7534f990ae504fa5aa364d6948a1760c" } modify_external_catalogs: true
I20240503 14:12:32.417753 30638 raft_consensus.cc:482] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.417963 30638 raft_consensus.cc:504] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.418709 30638 leader_election.cc:288] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.419350 28345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3ce05b422a4166bfac468acb84e4cc" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.419956 28150 leader_election.cc:341] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.421017 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 888636e9beca43edab96a8736751409e (table default.Table_a13cfbd07ce348578e9883c1205e8b9b [id=3854b08f7cd348c897aac4aacb7f5d06]). Expected version 3 got 0
I20240503 14:12:32.421399 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet b24ab3a5bba04a5092225a981f292222 (table default.Table_5354b912229e43539928c0e0df20a641 [id=294277efc25b401291a90d0fa70c90a9]). Expected version 3 got 0
I20240503 14:12:32.421677 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 5ac7b6df20b14d1ead036cdc42174d3a (table default.Table_bdce2849aeb843c0a01b36cd7dbcaee6 [id=0edef09212734f47be4ac77cca4fc186]). Expected version 2 got 0
I20240503 14:12:32.422948 30616 raft_consensus.cc:482] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.423128 30616 raft_consensus.cc:504] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.423926 30616 leader_election.cc:288] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.424825 28208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:32.425324 28208 raft_consensus.cc:2379] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a6b9b7913444d29c96581af6febf33 in current term 1: Already voted for candidate d472d9f94d8949cf86284d32a161cebc in this term.
I20240503 14:12:32.430318 30638 raft_consensus.cc:482] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.430538 30638 raft_consensus.cc:504] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.431357 30638 leader_election.cc:288] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.432015 28345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c17cfce7e644668817bf3483a098d4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.432879 28150 leader_election.cc:341] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:32.433898 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 58227us
I20240503 14:12:32.434813 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86451bc448428da7485e70e0d54b89 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cad03adb3ee419c962aaf20c72f8c04) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.435528 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86451bc448428da7485e70e0d54b89 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cad03adb3ee419c962aaf20c72f8c04) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:32.439136 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 7b86451bc448428da7485e70e0d54b89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b86451bc448428da7485e70e0d54b89 already in progress: creating tablet
I20240503 14:12:32.450049 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet e44f92cd95164ed09ec0865bb87a6123
I20240503 14:12:32.450953 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_85a520f4fd804e969bbb4149159976e1" } new_table_name: "default.Table_96c7f5ba8818439ea79ff4505e3871fb" modify_external_catalogs: true
I20240503 14:12:32.453809 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5fb312c322b34390a3823a7d55e21ebe" } new_table_name: "default.Table_a8c83eb7d38d4a3a97f85c95a1998213" modify_external_catalogs: true
I20240503 14:12:32.458489 30729 tablet_replica.cc:330] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:32.458767 30729 raft_consensus.cc:2229] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.458940 30729 raft_consensus.cc:2258] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.459515 30729 ts_tablet_manager.cc:1875] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.463080 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet e44f92cd95164ed09ec0865bb87a6123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:32.464205 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet e44f92cd95164ed09ec0865bb87a6123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e44f92cd95164ed09ec0865bb87a6123 already in progress: creating tablet
I20240503 14:12:32.463887 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet e44f92cd95164ed09ec0865bb87a6123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.357080 28604 tablet_bootstrap.cc:492] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:32.360999 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c79ffee3954b5c870f4a8aa6379517" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:32.472012 28489 raft_consensus.cc:3037] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.472012 28604 ts_tablet_manager.cc:1367] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.157s user 0.001s sys 0.005s
I20240503 14:12:32.472977 28604 raft_consensus.cc:348] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.473201 28604 raft_consensus.cc:374] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.473244 28604 raft_consensus.cc:729] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.473461 28604 consensus_queue.cc:260] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.474006 28604 ts_tablet_manager.cc:1398] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:32.474359 28604 tablet_bootstrap.cc:492] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:32.475404 28489 raft_consensus.cc:2454] T d7c79ffee3954b5c870f4a8aa6379517 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:32.420162 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3ce05b422a4166bfac468acb84e4cc" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.432159 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c17cfce7e644668817bf3483a098d4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.475919 28604 tablet_bootstrap.cc:654] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.441300 30716 raft_consensus.cc:482] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.476074 30716 raft_consensus.cc:504] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.476830 30716 leader_election.cc:288] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.476958 28290 leader_election.cc:302] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:32.476903 28661 log.cc:1196] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/34238c9859724d97a32f2a1d9315edb1
W20240503 14:12:32.477262 28150 leader_election.cc:341] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.477421 28150 leader_election.cc:302] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.477499 30616 raft_consensus.cc:2781] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:32.477625 30616 raft_consensus.cc:686] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:32.477721 30638 raft_consensus.cc:2726] T aa3ce05b422a4166bfac468acb84e4cc P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.477567 28661 ts_tablet_manager.cc:1909] T 34238c9859724d97a32f2a1d9315edb1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:32.477944 28150 leader_election.cc:341] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.478040 28150 leader_election.cc:302] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.477909 30616 consensus_queue.cc:237] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:32.478260 30638 raft_consensus.cc:2726] T a6c17cfce7e644668817bf3483a098d4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:32.439076 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 105791us
I20240503 14:12:32.483979 28457 tablet_service.cc:1514] Processing DeleteTablet for tablet e44f92cd95164ed09ec0865bb87a6123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:32.484405 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00dbc5a90674873b61b377e3e8d92fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:32.484745 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00dbc5a90674873b61b377e3e8d92fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.484797 28213 raft_consensus.cc:2454] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:32.487018 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86451bc448428da7485e70e0d54b89 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cad03adb3ee419c962aaf20c72f8c04) from {username='slave'} at 127.0.0.1:51314
W20240503 14:12:32.493468 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 312857b304e04b18b47a4ed6980a5cbf consensus state has no leader
I20240503 14:12:32.494061 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_76e967061ef74740937418c5be5d08a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.494784 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76e967061ef74740937418c5be5d08a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.497210 30638 raft_consensus.cc:482] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.497439 30638 raft_consensus.cc:504] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.498262 30638 leader_election.cc:288] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.499783 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dcf8bae3b5e4ff69a2afedb6c466ae4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.500461 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dcf8bae3b5e4ff69a2afedb6c466ae4" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.501192 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfc8ec1d6acb4127b7bf194226135b8c consensus state has no leader
W20240503 14:12:32.501183 28150 leader_election.cc:341] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:32.501580 28150 leader_election.cc:341] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.501672 28150 leader_election.cc:302] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.501657 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4a597910101d4133a1ad5201ce76c839" } modify_external_catalogs: true
I20240503 14:12:32.501875 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_4a597910101d4133a1ad5201ce76c839" } modify_external_catalogs: true
I20240503 14:12:32.503655 30638 raft_consensus.cc:2726] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.512677 28604 tablet_bootstrap.cc:492] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:32.512871 28604 ts_tablet_manager.cc:1367] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240503 14:12:32.513916 28604 raft_consensus.cc:348] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.514139 28604 raft_consensus.cc:374] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.518678 28033 catalog_manager.cc:5484] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:32.519234 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_ec97037ea5a849bfb48a72d6771fbf1f [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 2 got 0
I20240503 14:12:32.519976 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:32.514219 28604 raft_consensus.cc:729] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.523769 28604 consensus_queue.cc:260] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.524694 28604 ts_tablet_manager.cc:1398] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:12:32.525025 30716 raft_consensus.cc:482] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.525153 30716 raft_consensus.cc:504] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.525401 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d47ff25520da4d3c87c340cdaae46100" } modify_external_catalogs: true
I20240503 14:12:32.525579 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_d47ff25520da4d3c87c340cdaae46100" } modify_external_catalogs: true
I20240503 14:12:32.525856 30716 leader_election.cc:288] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.526119 28604 tablet_bootstrap.cc:492] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:32.529491 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 888636e9beca43edab96a8736751409e (table default.Table_a13cfbd07ce348578e9883c1205e8b9b [id=3854b08f7cd348c897aac4aacb7f5d06]). Expected version 3 got 0
I20240503 14:12:32.529845 28033 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:32.530361 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a83532bf2852473a9c51ce7cc78b5d34
I20240503 14:12:32.531100 28028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cafee16021144efc9522addcc165ad89" } modify_external_catalogs: true
I20240503 14:12:32.531313 28028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cafee16021144efc9522addcc165ad89" } modify_external_catalogs: true
I20240503 14:12:32.535423 28604 tablet_bootstrap.cc:654] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.479192 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1145) took 1826 ms. Trace:
I20240503 14:12:32.536011 28661 rpcz_store.cc:276] 0503 14:12:30.653180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.653232 (+ 52us) service_pool.cc:225] Handling call
0503 14:12:32.091540 (+1438308us) ts_tablet_manager.cc:1120] Deleting tablet 34238c9859724d97a32f2a1d9315edb1
0503 14:12:32.091555 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.323616 (+232061us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.353646 (+ 30030us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.479159 (+125513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409691}
I20240503 14:12:32.537739 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1143) took 1884 ms. Trace:
I20240503 14:12:32.540251 28661 rpcz_store.cc:276] 0503 14:12:30.653007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.653356 (+ 349us) service_pool.cc:225] Handling call
0503 14:12:32.537551 (+1884195us) ts_tablet_manager.cc:1120] Deleting tablet 471992c1f7e84ef1953cd85448a492ec
0503 14:12:32.537567 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.537714 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1855660}
I20240503 14:12:32.543954 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1144) took 1890 ms. Trace:
I20240503 14:12:32.545285 28335 tablet_service.cc:1467] Processing CreateTablet for tablet e89d3aa5a23341e8a2d051dfc78d8f1a (DEFAULT_TABLE table=default.Table_f6d60f02736f42f1a82698ec52941104 [id=812e9a2fa879420f8a90423574accec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.545827 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89d3aa5a23341e8a2d051dfc78d8f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.546012 28197 tablet_service.cc:1467] Processing CreateTablet for tablet 48854f5c3074416cbd0bd60b821c61ea (DEFAULT_TABLE table=default.Table_fc769de006724067ba44a724436e6a35 [id=dfd1b816d4684023928772ea7848dfe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.546417 28197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48854f5c3074416cbd0bd60b821c61ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.547470 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 836c131616c44feba2b39b26bf9d0b9e (DEFAULT_TABLE table=default.Table_2c0811e5c5b3481bbd30068c14bde1b9 [id=58e807f1e27540f98ed00b23aa79c1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.547880 28186 tablet_service.cc:1467] Processing CreateTablet for tablet e89d3aa5a23341e8a2d051dfc78d8f1a (DEFAULT_TABLE table=default.Table_f6d60f02736f42f1a82698ec52941104 [id=812e9a2fa879420f8a90423574accec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.547919 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 48854f5c3074416cbd0bd60b821c61ea (DEFAULT_TABLE table=default.Table_fc769de006724067ba44a724436e6a35 [id=dfd1b816d4684023928772ea7848dfe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.548416 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48854f5c3074416cbd0bd60b821c61ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.548419 28183 tablet_service.cc:1467] Processing CreateTablet for tablet de5bb8aa4fc8443bb5f0132c6fe4a33f (DEFAULT_TABLE table=default.Table_5ef9d168912f483c8004a5bbfe02e07d [id=a92b5ef677414af8aee512caad518546]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.548739 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5bb8aa4fc8443bb5f0132c6fe4a33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.549404 28187 tablet_service.cc:1467] Processing CreateTablet for tablet f0a7e0f4297f4b638466c51798feed45 (DEFAULT_TABLE table=default.Table_2f4bc37491894d27aae1c52cf3d365e9 [id=41c5cf44a03f467d9fe01b84b48af63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.549530 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836c131616c44feba2b39b26bf9d0b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.549928 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 27b25518570e4f2eb2666d420a9ef39c (DEFAULT_TABLE table=default.Table_c9fcd1ad02784a29858da8e74a9c9866 [id=38c77a1688424f44b3aeeb9384b445bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.550065 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89d3aa5a23341e8a2d051dfc78d8f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.550351 28196 tablet_service.cc:1467] Processing CreateTablet for tablet ddef09d920814354bf036d260da8b722 (DEFAULT_TABLE table=default.Table_2742208c2e8f4596952e282b1b29c727 [id=40fecf472adc47b68cd1b10dd99b071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.550638 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddef09d920814354bf036d260da8b722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.551985 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 3797fd0456554474953f11faa1736451 (DEFAULT_TABLE table=default.Table_e7c848ebbd6a4038a0404bddcb18f521 [id=e6700c57dc3c4efd845c92c77d794b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.552443 28180 tablet_service.cc:1467] Processing CreateTablet for tablet a58c1c4aba5d4d5c9798e0ce0ee12839 (DEFAULT_TABLE table=default.Table_decc999a1649405b8e7301584192c536 [id=ccb4698582a54cdcac8066d50c7af7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.553965 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 018abdeae16247408f09340f10d56ff6 (DEFAULT_TABLE table=default.Table_f2d1edcdf50346739221f0a19001dd4c [id=05b720f3753747c6b8278877ead49f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.545275 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeec26f579e4c43ade242f345b514a5 (DEFAULT_TABLE table=default.Table_a5bf58872e734741a8443b8a1a3f5424 [id=3aceb69bf08246628955eeb2e2f9f0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.554971 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b25518570e4f2eb2666d420a9ef39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.554908 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 836c131616c44feba2b39b26bf9d0b9e (DEFAULT_TABLE table=default.Table_2c0811e5c5b3481bbd30068c14bde1b9 [id=58e807f1e27540f98ed00b23aa79c1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.555140 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeec26f579e4c43ade242f345b514a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.555323 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836c131616c44feba2b39b26bf9d0b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.555562 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a7e0f4297f4b638466c51798feed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.555753 28329 tablet_service.cc:1467] Processing CreateTablet for tablet de5bb8aa4fc8443bb5f0132c6fe4a33f (DEFAULT_TABLE table=default.Table_5ef9d168912f483c8004a5bbfe02e07d [id=a92b5ef677414af8aee512caad518546]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.556135 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5bb8aa4fc8443bb5f0132c6fe4a33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.556406 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3797fd0456554474953f11faa1736451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.557392 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58c1c4aba5d4d5c9798e0ce0ee12839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.557870 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018abdeae16247408f09340f10d56ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.560725 28318 tablet_service.cc:1467] Processing CreateTablet for tablet f0a7e0f4297f4b638466c51798feed45 (DEFAULT_TABLE table=default.Table_2f4bc37491894d27aae1c52cf3d365e9 [id=41c5cf44a03f467d9fe01b84b48af63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.561228 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a7e0f4297f4b638466c51798feed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.563084 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 27b25518570e4f2eb2666d420a9ef39c (DEFAULT_TABLE table=default.Table_c9fcd1ad02784a29858da8e74a9c9866 [id=38c77a1688424f44b3aeeb9384b445bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.563923 28337 tablet_service.cc:1467] Processing CreateTablet for tablet a58c1c4aba5d4d5c9798e0ce0ee12839 (DEFAULT_TABLE table=default.Table_decc999a1649405b8e7301584192c536 [id=ccb4698582a54cdcac8066d50c7af7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.564182 28195 tablet_service.cc:1467] Processing CreateTablet for tablet d064d7ffb2654b239e6695af35108cf0 (DEFAULT_TABLE table=default.Table_d67437cde20547fb8188102d8a8e9d54 [id=1c47e63a5a104968a994768fd2b67c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.564239 28322 tablet_service.cc:1467] Processing CreateTablet for tablet ddef09d920814354bf036d260da8b722 (DEFAULT_TABLE table=default.Table_2742208c2e8f4596952e282b1b29c727 [id=40fecf472adc47b68cd1b10dd99b071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.564612 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddef09d920814354bf036d260da8b722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.564615 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d064d7ffb2654b239e6695af35108cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.566346 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58c1c4aba5d4d5c9798e0ce0ee12839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.567641 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b25518570e4f2eb2666d420a9ef39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.568425 28178 tablet_service.cc:1467] Processing CreateTablet for tablet 1b736bb7d13d4837af468a54208a4a75 (DEFAULT_TABLE table=default.Table_97a23538f3064d93a85e1b1218f01e9c [id=93178d6008234770bd8dcab5c797b068]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.568850 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b736bb7d13d4837af468a54208a4a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.570794 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2a3432263440f7a867dfb10249bc2d (DEFAULT_TABLE table=default.Table_1fbfaf1680ed48ef8c60061da7249ad9 [id=e8904211437b4d8c9d275cd69b2935d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.571224 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2a3432263440f7a867dfb10249bc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.571322 28600 tablet_bootstrap.cc:492] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.571456 28600 ts_tablet_manager.cc:1367] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.229s user 0.026s sys 0.026s
I20240503 14:12:32.572201 28600 raft_consensus.cc:348] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.572405 28600 raft_consensus.cc:374] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.572450 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 9897f05f81ef44e0ae272b11d7eaf1be (DEFAULT_TABLE table=default.Table_9099811cf7874977804669fff91540ca [id=9aee19c788de476f89647b5c515128a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.572876 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9897f05f81ef44e0ae272b11d7eaf1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.572451 28600 raft_consensus.cc:729] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.576606 28600 consensus_queue.cc:260] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.577558 28600 ts_tablet_manager.cc:1398] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:32.579032 28600 tablet_bootstrap.cc:492] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.582610 28600 tablet_bootstrap.cc:654] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.584260 30739 raft_consensus.cc:482] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.584452 30739 raft_consensus.cc:504] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.585265 30739 leader_election.cc:288] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.589059 28600 tablet_bootstrap.cc:492] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.589588 28600 ts_tablet_manager.cc:1367] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20240503 14:12:32.589730 28330 tablet_service.cc:1467] Processing CreateTablet for tablet 018abdeae16247408f09340f10d56ff6 (DEFAULT_TABLE table=default.Table_f2d1edcdf50346739221f0a19001dd4c [id=05b720f3753747c6b8278877ead49f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.590230 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018abdeae16247408f09340f10d56ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.590430 28600 raft_consensus.cc:348] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.590617 28600 raft_consensus.cc:374] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.590667 28600 raft_consensus.cc:729] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.590864 28600 consensus_queue.cc:260] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.591555 28600 ts_tablet_manager.cc:1398] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:32.591856 28600 tablet_bootstrap.cc:492] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.593489 28318 tablet_service.cc:1467] Processing CreateTablet for tablet d064d7ffb2654b239e6695af35108cf0 (DEFAULT_TABLE table=default.Table_d67437cde20547fb8188102d8a8e9d54 [id=1c47e63a5a104968a994768fd2b67c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.593808 28600 tablet_bootstrap.cc:654] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.593982 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d064d7ffb2654b239e6695af35108cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.593575 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2a3432263440f7a867dfb10249bc2d (DEFAULT_TABLE table=default.Table_1fbfaf1680ed48ef8c60061da7249ad9 [id=e8904211437b4d8c9d275cd69b2935d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.596652 28333 tablet_service.cc:1467] Processing CreateTablet for tablet 9897f05f81ef44e0ae272b11d7eaf1be (DEFAULT_TABLE table=default.Table_9099811cf7874977804669fff91540ca [id=9aee19c788de476f89647b5c515128a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.596966 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2a3432263440f7a867dfb10249bc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.597265 28326 tablet_service.cc:1467] Processing CreateTablet for tablet 1b736bb7d13d4837af468a54208a4a75 (DEFAULT_TABLE table=default.Table_97a23538f3064d93a85e1b1218f01e9c [id=93178d6008234770bd8dcab5c797b068]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.597618 28326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b736bb7d13d4837af468a54208a4a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.601368 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9897f05f81ef44e0ae272b11d7eaf1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.610796 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeec26f579e4c43ade242f345b514a5 (DEFAULT_TABLE table=default.Table_a5bf58872e734741a8443b8a1a3f5424 [id=3aceb69bf08246628955eeb2e2f9f0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.611351 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeec26f579e4c43ade242f345b514a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.617775 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 3797fd0456554474953f11faa1736451 (DEFAULT_TABLE table=default.Table_e7c848ebbd6a4038a0404bddcb18f521 [id=e6700c57dc3c4efd845c92c77d794b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.618381 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3797fd0456554474953f11faa1736451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.623970 28614 tablet_bootstrap.cc:492] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
W20240503 14:12:32.625311 28150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 123699us
I20240503 14:12:32.626396 30739 raft_consensus.cc:482] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.626576 30739 raft_consensus.cc:504] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.626667 28614 ts_tablet_manager.cc:1367] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.223s user 0.009s sys 0.003s
I20240503 14:12:32.627256 30739 leader_election.cc:288] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.627605 28614 raft_consensus.cc:348] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.627830 28614 raft_consensus.cc:374] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.627914 28614 raft_consensus.cc:729] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.628186 28614 consensus_queue.cc:260] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.628806 28614 ts_tablet_manager.cc:1398] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:32.629146 28614 tablet_bootstrap.cc:492] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.629156 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7ca19dd15f48a68ace7fefc7ae660f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.630828 28614 tablet_bootstrap.cc:654] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.578630 28604 tablet_bootstrap.cc:492] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:32.626363 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7ca19dd15f48a68ace7fefc7ae660f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.632524 28604 ts_tablet_manager.cc:1367] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.000s
I20240503 14:12:32.632354 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet a83532bf2852473a9c51ce7cc78b5d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.632920 28333 tablet_service.cc:1467] Processing CreateTablet for tablet ca020cc71952417d821b9c7a6cb7b356 (DEFAULT_TABLE table=default.Table_ee4900123ee64b90bfc6b5e340346bcf [id=9e91bec22b8d4ce2ab424b720a33f3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.633443 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca020cc71952417d821b9c7a6cb7b356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.633419 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b468dd4102495abd4337edcfc6340a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.633468 28604 raft_consensus.cc:348] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.633630 28604 raft_consensus.cc:374] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.633677 28604 raft_consensus.cc:729] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.633906 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b468dd4102495abd4337edcfc6340a" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.633942 28604 consensus_queue.cc:260] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.634615 28604 ts_tablet_manager.cc:1398] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:32.634918 28604 tablet_bootstrap.cc:492] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:32.637064 28604 tablet_bootstrap.cc:654] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:32.634614 28150 leader_election.cc:341] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:32.637534 28150 leader_election.cc:341] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:32.637823 28150 leader_election.cc:341] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.637959 28150 leader_election.cc:302] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.638288 30739 raft_consensus.cc:2726] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:32.638788 28150 leader_election.cc:341] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.638967 28150 leader_election.cc:302] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.639222 30739 raft_consensus.cc:2726] T a6b468dd4102495abd4337edcfc6340a P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.642376 28661 rpcz_store.cc:276] 0503 14:12:30.653084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.653472 (+ 388us) service_pool.cc:225] Handling call
0503 14:12:32.541771 (+1888299us) ts_tablet_manager.cc:1120] Deleting tablet 9cad0a4cdfa8471bbd7066b91e85022e
0503 14:12:32.541784 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.543915 (+ 2131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859839}
I20240503 14:12:32.647615 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 77d8ce2e91b74293a2652ab3d4bee82d (DEFAULT_TABLE table=default.Table_390a8ae260bb40809e52f71153fa46f0 [id=02a195b040b24f7981ad3d6f42edb9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.648283 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet a83532bf2852473a9c51ce7cc78b5d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.648962 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d8ce2e91b74293a2652ab3d4bee82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.650003 28180 tablet_service.cc:1467] Processing CreateTablet for tablet ee0897770c0a4a38b6a77624f630f722 (DEFAULT_TABLE table=default.Table_02b32574c2ac4fdab08b851ecd21084e [id=2ce3f18a2b1443bdb94dac7c6b44abc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.650599 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 77d8ce2e91b74293a2652ab3d4bee82d (DEFAULT_TABLE table=default.Table_390a8ae260bb40809e52f71153fa46f0 [id=02a195b040b24f7981ad3d6f42edb9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.651095 28196 tablet_service.cc:1467] Processing CreateTablet for tablet ca020cc71952417d821b9c7a6cb7b356 (DEFAULT_TABLE table=default.Table_ee4900123ee64b90bfc6b5e340346bcf [id=9e91bec22b8d4ce2ab424b720a33f3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.654727 28600 tablet_bootstrap.cc:492] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.655680 28600 ts_tablet_manager.cc:1367] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.003s
I20240503 14:12:32.656351 28321 tablet_service.cc:1467] Processing CreateTablet for tablet ee0897770c0a4a38b6a77624f630f722 (DEFAULT_TABLE table=default.Table_02b32574c2ac4fdab08b851ecd21084e [id=2ce3f18a2b1443bdb94dac7c6b44abc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.656520 28600 raft_consensus.cc:348] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.656805 28600 raft_consensus.cc:374] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.656893 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0897770c0a4a38b6a77624f630f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.656904 28600 raft_consensus.cc:729] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.657182 28600 consensus_queue.cc:260] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.657891 28600 ts_tablet_manager.cc:1398] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 14:12:32.631394 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 149130us
W20240503 14:12:32.660492 28427 consensus_peers.cc:485] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:32.660871 28427 consensus_peers.cc:485] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:32.661248 28427 consensus_peers.cc:485] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:32.642974 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1149) took 1847 ms. Trace:
I20240503 14:12:32.664678 28661 rpcz_store.cc:276] 0503 14:12:30.795682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:30.796077 (+ 395us) service_pool.cc:225] Handling call
0503 14:12:32.642792 (+1846715us) ts_tablet_manager.cc:1120] Deleting tablet aad397ed76ea4cf8820312ba1362512f
0503 14:12:32.642807 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.642946 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846481}
I20240503 14:12:32.665551 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1160) took 1549 ms. Trace:
I20240503 14:12:32.665812 28661 rpcz_store.cc:276] 0503 14:12:31.116449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.116508 (+ 59us) service_pool.cc:225] Handling call
0503 14:12:32.665303 (+1548795us) ts_tablet_manager.cc:1120] Deleting tablet 62e4f6e3908e420e9fa156ee24b437d6
0503 14:12:32.665343 (+ 40us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.665504 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548514}
I20240503 14:12:32.666430 30739 raft_consensus.cc:482] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.666595 30739 raft_consensus.cc:504] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.666711 28661 tablet_replica.cc:330] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:32.667212 28661 raft_consensus.cc:2229] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:32.667330 30739 leader_election.cc:288] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.668481 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007c84fd9d9247728b0babd48689aba8" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.669157 28150 leader_election.cc:341] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:32.669936 28018 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet bfc8ec1d6acb4127b7bf194226135b8c (table default.Table_96c7f5ba8818439ea79ff4505e3871fb [id=ed81e75ce5ae45a2ac9a547c1903c681], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bfc8ec1d6acb4127b7bf194226135b8c consensus state has no leader
I20240503 14:12:32.670540 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f37ea39f88b04f0abb0445d4342406c3
I20240503 14:12:32.671412 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_0d432ccf7afb4aa99befb2d05817d6ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:32.673069 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007c84fd9d9247728b0babd48689aba8" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:32.672063 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d432ccf7afb4aa99befb2d05817d6ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.672363 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 137c9eef465843049e93374367e562af
I20240503 14:12:32.677755 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet a05f786073084bd1b8c472c20dc8bff6 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:32.672504 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 2c8e6e922f5b4577861af5bdbee94f72
I20240503 14:12:32.678755 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_89a641dd82434ebeb2c8ed285b1913ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.679031 28150 leader_election.cc:341] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.679169 28150 leader_election.cc:302] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.679275 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0897770c0a4a38b6a77624f630f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.679337 30729 ts_tablet_manager.cc:1888] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:32.679353 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89a641dd82434ebeb2c8ed285b1913ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.679438 30729 log.cc:1196] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7b86451bc448428da7485e70e0d54b89
I20240503 14:12:32.679483 30739 raft_consensus.cc:2726] T 007c84fd9d9247728b0babd48689aba8 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.679859 30729 ts_tablet_manager.cc:1909] T 7b86451bc448428da7485e70e0d54b89 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:32.680207 28600 tablet_bootstrap.cc:492] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.680285 28661 pending_rounds.cc:62] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Trying to abort 1 pending ops.
I20240503 14:12:32.680372 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d8ce2e91b74293a2652ab3d4bee82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.680354 28661 pending_rounds.cc:68] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597775089348608 op_type: NO_OP noop_request { }
I20240503 14:12:32.680476 28661 raft_consensus.cc:2866] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:32.680572 28661 raft_consensus.cc:2258] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.683220 28600 tablet_bootstrap.cc:654] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.684146 28661 ts_tablet_manager.cc:1875] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.684520 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca020cc71952417d821b9c7a6cb7b356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.688246 30446 raft_consensus.cc:482] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.688426 30446 raft_consensus.cc:504] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:32.689108 30446 leader_election.cc:288] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.424547 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f223cbbcaa5402a888b084df0fe1c27" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.690536 28290 leader_election.cc:302] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.690855 30616 raft_consensus.cc:3037] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.693572 30616 raft_consensus.cc:2726] T 0f223cbbcaa5402a888b084df0fe1c27 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:32.694458 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a83532bf2852473a9c51ce7cc78b5d34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a83532bf2852473a9c51ce7cc78b5d34 already in progress: creating tablet
I20240503 14:12:32.695374 28614 tablet_bootstrap.cc:492] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.695652 28614 ts_tablet_manager.cc:1367] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240503 14:12:32.696508 28614 raft_consensus.cc:348] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.696904 28614 raft_consensus.cc:374] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.697029 28614 raft_consensus.cc:729] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.697368 28614 consensus_queue.cc:260] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.698577 28614 ts_tablet_manager.cc:1398] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:32.699625 28614 tablet_bootstrap.cc:492] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.704030 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet f37ea39f88b04f0abb0445d4342406c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.704284 28614 tablet_bootstrap.cc:654] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.709287 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 137c9eef465843049e93374367e562af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.709237 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7b86451bc448428da7485e70e0d54b89 (table default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]) successfully deleted
I20240503 14:12:32.709233 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8e6e922f5b4577861af5bdbee94f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58028
W20240503 14:12:32.709578 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet e44f92cd95164ed09ec0865bb87a6123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e44f92cd95164ed09ec0865bb87a6123 already in progress: creating tablet
I20240503 14:12:32.709894 28600 tablet_bootstrap.cc:492] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.710131 28600 ts_tablet_manager.cc:1367] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20240503 14:12:32.710575 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet 137c9eef465843049e93374367e562af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.710808 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet f37ea39f88b04f0abb0445d4342406c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.711006 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8e6e922f5b4577861af5bdbee94f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.713425 28600 raft_consensus.cc:348] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.713960 28600 raft_consensus.cc:374] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.714082 28600 raft_consensus.cc:729] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.714448 28600 consensus_queue.cc:260] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.715745 28600 ts_tablet_manager.cc:1398] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:32.716042 28263 heartbeater.cc:347] Connected to a master server at 127.27.50.254:36571
W20240503 14:12:32.717844 28427 leader_election.cc:341] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:32.721248 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet a83532bf2852473a9c51ce7cc78b5d34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a83532bf2852473a9c51ce7cc78b5d34 already in progress: creating tablet
W20240503 14:12:32.722009 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 137c9eef465843049e93374367e562af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 137c9eef465843049e93374367e562af already in progress: creating tablet
I20240503 14:12:32.723116 28600 tablet_bootstrap.cc:492] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.725584 28600 tablet_bootstrap.cc:654] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.726182 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet a05f786073084bd1b8c472c20dc8bff6
I20240503 14:12:32.726696 28029 catalog_manager.cc:6554] ReplaceTablet: tablet a05f786073084bd1b8c472c20dc8bff6 deleted and replaced by tablet 2b828eb16891434f9031b9e4d29b18d8
I20240503 14:12:32.726938 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet 77f6fb009bdd4f70ba90e7b5f972cea9 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:32.727427 28029 master_service.cc:946] ReplaceTablet: received request to replace tablet f7d18fdc987d4879a642438e748eeb62 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:32.731251 28661 ts_tablet_manager.cc:1888] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:32.731396 28661 log.cc:1196] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/0c7779f7a9544e17aea7135eab2ccd5f
I20240503 14:12:32.731772 28661 ts_tablet_manager.cc:1909] T 0c7779f7a9544e17aea7135eab2ccd5f P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:32.732599 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3f049487b1314f94877e91d7f1a40916"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.733304 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f049487b1314f94877e91d7f1a40916 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.735247 28604 tablet_bootstrap.cc:492] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:32.735435 28604 ts_tablet_manager.cc:1367] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.101s user 0.000s sys 0.004s
I20240503 14:12:32.736567 28604 raft_consensus.cc:348] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.736795 28604 raft_consensus.cc:374] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.736847 28604 raft_consensus.cc:729] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.736917 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 780b53efefcf42569bba2fcabd21e211 (table default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c]). Expected version 2 got 0
I20240503 14:12:32.737146 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 725b4c9a47c0458ba188c0cf62f3fe48 (table default.Table_35e27f7a242d4ffaa19a6e07ec525559 [id=ec20cb1644174d6c8a21107acbf0a040]). Expected version 1 got 0
I20240503 14:12:32.737093 28604 consensus_queue.cc:260] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.737766 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet a703cfbaad2a4a618abb9e07b4930cee (table default.Table_692756cbec294f19a8cd798b13128ca4 [id=16cce63b7e0b4afbacc9feffaa625932]). Expected version 2 got 0
I20240503 14:12:32.738440 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 99aa7e59c45c45c7bc1f726c97426b89 (table default.Table_a5de24e133584200af645a728b1346e1 [id=8894157a06784f4288235959f27849b8]). Expected version 2 got 0
I20240503 14:12:32.738616 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 788e4bfa1e414630939bc19da1ede3fb (table default.Table_b7b3cecd524e40feb9c6251d0403b764 [id=ddef27f860534921adab476e0f391782]). Expected version 2 got 0
I20240503 14:12:32.738934 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 2f7b805aee554f8fa4c198ce3409c6e6 (table default.Table_ee9f2d2bd2b54111963a23deaf110216 [id=4df57b62a5b34ac182e799495a848041]). Expected version 1 got 0
I20240503 14:12:32.739105 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_ec97037ea5a849bfb48a72d6771fbf1f [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 2 got 0
I20240503 14:12:32.739297 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 888636e9beca43edab96a8736751409e (table default.Table_a13cfbd07ce348578e9883c1205e8b9b [id=3854b08f7cd348c897aac4aacb7f5d06]). Expected version 3 got 0
I20240503 14:12:32.739423 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet b24ab3a5bba04a5092225a981f292222 (table default.Table_5354b912229e43539928c0e0df20a641 [id=294277efc25b401291a90d0fa70c90a9]). Expected version 3 got 0
I20240503 14:12:32.739548 28027 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 5ac7b6df20b14d1ead036cdc42174d3a (table default.Table_bdce2849aeb843c0a01b36cd7dbcaee6 [id=0edef09212734f47be4ac77cca4fc186]). Expected version 2 got 0
I20240503 14:12:32.742635 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_99031ccd4e8e4d1c9518552fdb4c2a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.743321 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99031ccd4e8e4d1c9518552fdb4c2a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.747928 28026 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_ca6cee392d934dd3ac773e100ae031d2 [id=32be054076954aed9265888ca62e510b] alter complete (version 4)
I20240503 14:12:32.751453 28032 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_831065d014d241bfbb471ac2cfd336bd [id=566ce63bc6594fb29df2f8eb42c940e2] alter complete (version 3)
I20240503 14:12:32.751411 28604 ts_tablet_manager.cc:1398] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
W20240503 14:12:32.753103 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 788e4bfa1e414630939bc19da1ede3fb consensus state has no leader
I20240503 14:12:32.753321 28031 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33344 (request call id 1278) took 1871 ms. Trace:
I20240503 14:12:32.753413 28031 rpcz_store.cc:276] 0503 14:12:30.881558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.248995 (+367437us) service_pool.cc:225] Handling call
0503 14:12:31.249502 (+ 507us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0503 14:12:31.249743 (+ 241us) catalog_manager.cc:3613] Apply alter schema
0503 14:12:31.252764 (+ 3021us) catalog_manager.cc:3636] Acquired catalog manager lock
0503 14:12:31.252862 (+ 98us) catalog_manager.cc:3783] Updating metadata on disk
0503 14:12:31.318973 (+ 66111us) catalog_manager.cc:3814] Committing alterations to in-memory state
0503 14:12:32.753283 (+1434310us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:31.253515 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:31.253644 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:31.253651 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:31.253653 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:31.253772 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:31.253786 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:31.253790 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:31.253861 (+ 71us) tablet.cc:699] Row locks acquired
0503 14:12:31.253863 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:31.253907 (+ 44us) write_op.cc:270] Start()
0503 14:12:31.253943 (+ 36us) write_op.cc:276] Timestamp: P: 1714745551253866 usec, L: 0
0503 14:12:31.253948 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:31.254145 (+ 197us) log.cc:842] Serialized 405 byte log entry
0503 14:12:31.315997 (+ 61852us) op_driver.cc:444] REPLICATION: finished
0503 14:12:31.317403 (+ 1406us) write_op.cc:301] APPLY: starting
0503 14:12:31.317415 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:31.317445 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:31.317469 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:31.317511 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:31.317537 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:31.317544 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:31.317608 (+ 64us) log.cc:842] Serialized 23 byte log entry
0503 14:12:31.317665 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:31.317679 (+ 14us) write_op.cc:454] Released schema lock
0503 14:12:31.317684 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:31.317739 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:31.317743 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1372,"apply.run_cpu_time_us":387,"apply.run_wall_time_us":387,"num_ops":1,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":740,"prepare.run_wall_time_us":740,"replication_time_us":62033}]]}
I20240503 14:12:32.754318 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_196b6e1c710b483889aef9e34ab1d8fd" } new_table_name: "default.Table_20d5f53b083e4b2d8bdb88b3ef0e73f5" modify_external_catalogs: true
I20240503 14:12:32.766110 30748 tablet.cc:1722] T 780b53efefcf42569bba2fcabd21e211 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:32.766199 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f37ea39f88b04f0abb0445d4342406c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f37ea39f88b04f0abb0445d4342406c3 already in progress: creating tablet
I20240503 14:12:32.767922 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f5103b31ed45b480252dcf94b72f9c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.769178 28604 tablet_bootstrap.cc:492] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:32.772501 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 137c9eef465843049e93374367e562af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 137c9eef465843049e93374367e562af already in progress: creating tablet
W20240503 14:12:32.775699 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 2c8e6e922f5b4577861af5bdbee94f72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8e6e922f5b4577861af5bdbee94f72 already in progress: creating tablet
I20240503 14:12:32.777673 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebec4622ef4950bd1c70451958c59b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.778612 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet a05f786073084bd1b8c472c20dc8bff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b828eb16891434f9031b9e4d29b18d8) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.779417 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 34238c9859724d97a32f2a1d9315edb1 (table default.Table_6a18ec9a972b4ad99771cd58f58c1a5b [id=fdbf87909c2940679817af17da94d101]) successfully deleted
I20240503 14:12:32.779136 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet a05f786073084bd1b8c472c20dc8bff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b828eb16891434f9031b9e4d29b18d8) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:32.779693 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 471992c1f7e84ef1953cd85448a492ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 471992c1f7e84ef1953cd85448a492ec already in progress: creating tablet
W20240503 14:12:32.779907 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9cad0a4cdfa8471bbd7066b91e85022e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cad0a4cdfa8471bbd7066b91e85022e already in progress: creating tablet
W20240503 14:12:32.780134 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet aad397ed76ea4cf8820312ba1362512f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aad397ed76ea4cf8820312ba1362512f already in progress: creating tablet
W20240503 14:12:32.780324 28017 catalog_manager.cc:4797] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 62e4f6e3908e420e9fa156ee24b437d6 because the tablet was not found. No further retry: Not found: Tablet not found: 62e4f6e3908e420e9fa156ee24b437d6
I20240503 14:12:32.780299 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebec4622ef4950bd1c70451958c59b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
W20240503 14:12:32.780498 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f37ea39f88b04f0abb0445d4342406c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f37ea39f88b04f0abb0445d4342406c3 already in progress: creating tablet
I20240503 14:12:32.780524 28212 raft_consensus.cc:2454] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:32.780665 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet 2c8e6e922f5b4577861af5bdbee94f72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8e6e922f5b4577861af5bdbee94f72 already in progress: creating tablet
I20240503 14:12:32.782522 28604 tablet_bootstrap.cc:654] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.785271 30558 raft_consensus.cc:482] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.785441 30558 raft_consensus.cc:504] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.786091 30558 leader_election.cc:288] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.779448 28614 tablet_bootstrap.cc:492] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.789808 28614 ts_tablet_manager.cc:1367] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
W20240503 14:12:32.789930 28427 consensus_peers.cc:485] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cdc868ddf0a490292eae78ebc8c8ec6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:32.790943 28600 tablet_bootstrap.cc:492] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.790971 28024 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_e3e45b04f7fa4b4ab619f2ee82a626cb [id=f7e1e9a608664cd6967ef0521d9cb671] alter complete (version 1)
I20240503 14:12:32.791239 28600 ts_tablet_manager.cc:1367] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.011s
I20240503 14:12:32.791427 28614 raft_consensus.cc:348] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.791652 28614 raft_consensus.cc:374] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.791719 28614 raft_consensus.cc:729] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.791962 28614 consensus_queue.cc:260] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.792536 28600 raft_consensus.cc:348] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.792788 28600 raft_consensus.cc:374] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.792879 28600 raft_consensus.cc:729] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.793159 28600 consensus_queue.cc:260] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.794206 28600 ts_tablet_manager.cc:1398] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:32.795403 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f7d18fdc987d4879a642438e748eeb62
I20240503 14:12:32.795679 28614 ts_tablet_manager.cc:1398] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:32.795959 28029 catalog_manager.cc:6554] ReplaceTablet: tablet f7d18fdc987d4879a642438e748eeb62 deleted and replaced by tablet ddc24f19b95e44f48f76fea5a22ba1c1
I20240503 14:12:32.796449 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_6eb30e0811914f60b33aa8ba9e286f8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.797068 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb30e0811914f60b33aa8ba9e286f8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.798673 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet bbc90d9b37bc4c57af3d29ce0f7da287 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:32.800113 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1161) took 1682 ms. Trace:
I20240503 14:12:32.800297 28030 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_3838859d5d884075bc288faedfd9ce98" } new_table_name: "default.Table_250dfe67eb1440168bc305601c7ed3f7" modify_external_catalogs: true
I20240503 14:12:32.800283 28661 rpcz_store.cc:276] 0503 14:12:31.117195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.117245 (+ 50us) service_pool.cc:225] Handling call
0503 14:12:32.666613 (+1549368us) ts_tablet_manager.cc:1120] Deleting tablet 0c7779f7a9544e17aea7135eab2ccd5f
0503 14:12:32.666627 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.702986 (+ 36359us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.731084 (+ 28098us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.800059 (+ 68975us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12533,"tablet-delete.queue_time_us":1548968}
I20240503 14:12:32.802130 28026 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_9f2d3cc9c6724dab9ba35ff24e85c236 [id=c358e28088d44cf598c96a1bd2a4321e] alter complete (version 1)
I20240503 14:12:32.803244 28661 tablet_replica.cc:330] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:32.803512 28661 raft_consensus.cc:2229] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
W20240503 14:12:32.803557 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5a62a4abd9d4c8dbf642364faad8539 consensus state has no leader
I20240503 14:12:32.803890 28661 raft_consensus.cc:2258] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.803998 28031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_cc70a5c7175f497ba45f4fcd8a950c9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:32.803588 28427 leader_election.cc:302] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
W20240503 14:12:32.804669 28031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc70a5c7175f497ba45f4fcd8a950c9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.805123 30558 raft_consensus.cc:2781] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:32.805218 30558 raft_consensus.cc:482] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:32.805370 30558 raft_consensus.cc:3037] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.808642 30558 raft_consensus.cc:504] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.809412 30558 leader_election.cc:288] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.814227 30616 raft_consensus.cc:482] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.814419 30616 raft_consensus.cc:504] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.815169 30616 leader_election.cc:288] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.815754 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59945d011b64cba87c854e2ffde8226" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.817349 28661 ts_tablet_manager.cc:1875] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.817873 30752 tablet.cc:1722] T 780b53efefcf42569bba2fcabd21e211 P d3a6b9b7913444d29c96581af6febf33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:32.819072 29828 tablet_replica.cc:330] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:32.819290 29828 raft_consensus.cc:2229] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.819470 29828 raft_consensus.cc:2258] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.820292 29828 ts_tablet_manager.cc:1875] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.824805 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d18fdc987d4879a642438e748eeb62 with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24f19b95e44f48f76fea5a22ba1c1) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.825242 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d18fdc987d4879a642438e748eeb62 with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24f19b95e44f48f76fea5a22ba1c1) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:32.829154 28290 leader_election.cc:341] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: b59945d011b64cba87c854e2ffde8226
I20240503 14:12:32.829315 28290 leader_election.cc:302] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.829751 30616 raft_consensus.cc:2726] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:32.831354 28427 consensus_peers.cc:485] T b59945d011b64cba87c854e2ffde8226 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b59945d011b64cba87c854e2ffde8226. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:32.834010 30729 tablet_replica.cc:330] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:32.834272 28600 tablet_bootstrap.cc:492] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.834282 30729 raft_consensus.cc:2229] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.834712 30729 raft_consensus.cc:2258] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.835378 30729 ts_tablet_manager.cc:1875] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.836163 28600 tablet_bootstrap.cc:654] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:32.835809 28427 leader_election.cc:341] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: d6f5103b31ed45b480252dcf94b72f9c
I20240503 14:12:32.839543 28427 leader_election.cc:302] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:32.841423 30446 raft_consensus.cc:2781] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:32.841553 30446 raft_consensus.cc:482] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:32.841626 30446 raft_consensus.cc:3037] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:32.842294 28024 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_bfff85704d88463383cbf94f9b121359 [id=bd1950deb4274d86bf288d501cf5c710] alter complete (version 2)
I20240503 14:12:32.843451 29828 ts_tablet_manager.cc:1888] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240503 14:12:32.843744 29828 log.cc:1196] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a05f786073084bd1b8c472c20dc8bff6
I20240503 14:12:32.845563 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet bbc90d9b37bc4c57af3d29ce0f7da287
I20240503 14:12:32.846033 28028 catalog_manager.cc:6554] ReplaceTablet: tablet bbc90d9b37bc4c57af3d29ce0f7da287 deleted and replaced by tablet ba0e8d36b7bf4bdb8111673121e2bfbe
I20240503 14:12:32.846448 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_f75dd17eefb64460ad4afba77ed17b65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:32.846910 30446 raft_consensus.cc:504] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
W20240503 14:12:32.847071 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f75dd17eefb64460ad4afba77ed17b65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.847491 28614 tablet_bootstrap.cc:492] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.847774 30446 leader_election.cc:288] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:32.849372 28614 tablet_bootstrap.cc:654] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.849509 28661 ts_tablet_manager.cc:1888] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:32.849613 28661 log.cc:1196] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/5cdc868ddf0a490292eae78ebc8c8ec6
W20240503 14:12:32.849627 28024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 118) took 927 ms (client timeout 1000 ms). Trace:
I20240503 14:12:32.849951 28661 ts_tablet_manager.cc:1909] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:32.849731 28024 rpcz_store.cc:269] 0503 14:12:31.922409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.986303 (+ 63894us) service_pool.cc:225] Handling call
0503 14:12:32.849600 (+863297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:32.759217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:32.759330 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:32.759337 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:32.759340 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:32.759448 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:32.759460 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:32.759466 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:32.759526 (+ 60us) tablet.cc:699] Row locks acquired
0503 14:12:32.759529 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:32.759564 (+ 35us) write_op.cc:270] Start()
0503 14:12:32.759597 (+ 33us) write_op.cc:276] Timestamp: P: 1714745552759430 usec, L: 0
0503 14:12:32.759600 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:32.759781 (+ 181us) log.cc:842] Serialized 356 byte log entry
0503 14:12:32.788271 (+ 28490us) op_driver.cc:444] REPLICATION: finished
0503 14:12:32.790549 (+ 2278us) write_op.cc:301] APPLY: starting
0503 14:12:32.790560 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:32.790587 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:32.790607 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:32.790647 (+ 40us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:32.790673 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:32.790680 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:32.790726 (+ 46us) log.cc:842] Serialized 23 byte log entry
0503 14:12:32.790763 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.790774 (+ 11us) write_op.cc:454] Released schema lock
0503 14:12:32.790779 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:32.790830 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.790833 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0503 14:12:32.806162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:32.806290 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:32.806297 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:32.806299 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:32.806456 (+ 157us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:32.806472 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:32.806477 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:32.806548 (+ 71us) tablet.cc:699] Row locks acquired
0503 14:12:32.806550 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:32.806595 (+ 45us) write_op.cc:270] Start()
0503 14:12:32.806637 (+ 42us) write_op.cc:276] Timestamp: P: 1714745552806455 usec, L: 0
0503 14:12:32.806640 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:32.806883 (+ 243us) log.cc:842] Serialized 356 byte log entry
0503 14:12:32.841320 (+ 34437us) op_driver.cc:444] REPLICATION: finished
0503 14:12:32.841648 (+ 328us) write_op.cc:301] APPLY: starting
0503 14:12:32.841688 (+ 40us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:32.841739 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:32.841774 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:32.841851 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:32.841891 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:32.841899 (+ 8us) write_op.cc:312] APPLY: finished
0503 14:12:32.841995 (+ 96us) log.cc:842] Serialized 23 byte log entry
0503 14:12:32.842075 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.842099 (+ 24us) write_op.cc:454] Released schema lock
0503 14:12:32.842108 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:32.842179 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.842183 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":20,"spinlock_wait_cycles":19840,"child_traces":[["op",{"apply.queue_time_us":2247,"apply.run_cpu_time_us":324,"apply.run_wall_time_us":324,"mutex_wait_us":1,"num_ops":1,"prepare.queue_time_us":10077,"prepare.run_cpu_time_us":624,"prepare.run_wall_time_us":622,"replication_time_us":28658}],["op",{"apply.queue_time_us":269,"apply.run_cpu_time_us":608,"apply.run_wall_time_us":607,"num_ops":1,"prepare.queue_time_us":13364,"prepare.run_cpu_time_us":809,"prepare.run_wall_time_us":809,"replication_time_us":34641}]]}
W20240503 14:12:32.852960 28030 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e644199208a45db99c085cd356a0584 consensus state has no leader
I20240503 14:12:32.853457 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_33f75df58c1a416c8ed8bf6247b0b31c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.854091 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33f75df58c1a416c8ed8bf6247b0b31c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.848227 28026 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_16583130ff3c4792a96fb0039f73843f [id=134086bbf5ef4b5e890cab53615e449e] alter complete (version 2)
I20240503 14:12:32.857172 30729 ts_tablet_manager.cc:1888] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240503 14:12:32.857841 30729 log.cc:1196] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a05f786073084bd1b8c472c20dc8bff6
I20240503 14:12:32.858309 30729 ts_tablet_manager.cc:1909] T a05f786073084bd1b8c472c20dc8bff6 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:32.859575 29828 ts_tablet_manager.cc:1909] T a05f786073084bd1b8c472c20dc8bff6 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:32.861387 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1162) took 1743 ms. Trace:
I20240503 14:12:32.861431 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_5ea81f41489c4dd0aa09e24ca08d7431"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.862047 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ea81f41489c4dd0aa09e24ca08d7431 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.862187 28661 rpcz_store.cc:276] 0503 14:12:31.117786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.117833 (+ 47us) service_pool.cc:225] Handling call
0503 14:12:32.803164 (+1685331us) ts_tablet_manager.cc:1120] Deleting tablet 5cdc868ddf0a490292eae78ebc8c8ec6
0503 14:12:32.803178 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:32.833390 (+ 30212us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.849386 (+ 15996us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:32.861355 (+ 11969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1685086}
I20240503 14:12:32.862556 28661 tablet_replica.cc:330] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:32.862756 28661 raft_consensus.cc:2229] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:32.864732 28661 raft_consensus.cc:2258] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.855998 28024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b65e95f8090a40e5abb366c2aa904d60" } modify_external_catalogs: true
I20240503 14:12:32.865499 28024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_b65e95f8090a40e5abb366c2aa904d60" } modify_external_catalogs: true
I20240503 14:12:32.880915 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeec26f579e4c43ade242f345b514a5 (DEFAULT_TABLE table=default.Table_a5bf58872e734741a8443b8a1a3f5424 [id=3aceb69bf08246628955eeb2e2f9f0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.881448 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeec26f579e4c43ade242f345b514a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.884716 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 48854f5c3074416cbd0bd60b821c61ea (DEFAULT_TABLE table=default.Table_fc769de006724067ba44a724436e6a35 [id=dfd1b816d4684023928772ea7848dfe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.885211 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48854f5c3074416cbd0bd60b821c61ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.886901 28614 tablet_bootstrap.cc:492] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.887073 28614 ts_tablet_manager.cc:1367] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20240503 14:12:32.887890 28614 raft_consensus.cc:348] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.888121 28614 raft_consensus.cc:374] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.888176 28614 raft_consensus.cc:729] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.888418 28461 tablet_service.cc:1467] Processing CreateTablet for tablet e89d3aa5a23341e8a2d051dfc78d8f1a (DEFAULT_TABLE table=default.Table_f6d60f02736f42f1a82698ec52941104 [id=812e9a2fa879420f8a90423574accec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.888404 28614 consensus_queue.cc:260] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.888854 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89d3aa5a23341e8a2d051dfc78d8f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.888990 28614 ts_tablet_manager.cc:1398] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:32.889240 28462 tablet_service.cc:1467] Processing CreateTablet for tablet ddef09d920814354bf036d260da8b722 (DEFAULT_TABLE table=default.Table_2742208c2e8f4596952e282b1b29c727 [id=40fecf472adc47b68cd1b10dd99b071c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.889588 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddef09d920814354bf036d260da8b722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.891960 28614 tablet_bootstrap.cc:492] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:32.892426 28464 tablet_service.cc:1467] Processing CreateTablet for tablet a58c1c4aba5d4d5c9798e0ce0ee12839 (DEFAULT_TABLE table=default.Table_decc999a1649405b8e7301584192c536 [id=ccb4698582a54cdcac8066d50c7af7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.892663 28466 tablet_service.cc:1467] Processing CreateTablet for tablet de5bb8aa4fc8443bb5f0132c6fe4a33f (DEFAULT_TABLE table=default.Table_5ef9d168912f483c8004a5bbfe02e07d [id=a92b5ef677414af8aee512caad518546]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.892832 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58c1c4aba5d4d5c9798e0ce0ee12839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.893018 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5bb8aa4fc8443bb5f0132c6fe4a33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.896415 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 018abdeae16247408f09340f10d56ff6 (DEFAULT_TABLE table=default.Table_f2d1edcdf50346739221f0a19001dd4c [id=05b720f3753747c6b8278877ead49f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.896486 28469 tablet_service.cc:1467] Processing CreateTablet for tablet f0a7e0f4297f4b638466c51798feed45 (DEFAULT_TABLE table=default.Table_2f4bc37491894d27aae1c52cf3d365e9 [id=41c5cf44a03f467d9fe01b84b48af63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.896838 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018abdeae16247408f09340f10d56ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.897069 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 27b25518570e4f2eb2666d420a9ef39c (DEFAULT_TABLE table=default.Table_c9fcd1ad02784a29858da8e74a9c9866 [id=38c77a1688424f44b3aeeb9384b445bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.897468 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b25518570e4f2eb2666d420a9ef39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.898218 28614 tablet_bootstrap.cc:654] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.900430 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 3797fd0456554474953f11faa1736451 (DEFAULT_TABLE table=default.Table_e7c848ebbd6a4038a0404bddcb18f521 [id=e6700c57dc3c4efd845c92c77d794b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.900813 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3797fd0456554474953f11faa1736451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.892438 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 836c131616c44feba2b39b26bf9d0b9e (DEFAULT_TABLE table=default.Table_2c0811e5c5b3481bbd30068c14bde1b9 [id=58e807f1e27540f98ed00b23aa79c1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.901623 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836c131616c44feba2b39b26bf9d0b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.902897 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 1b736bb7d13d4837af468a54208a4a75 (DEFAULT_TABLE table=default.Table_97a23538f3064d93a85e1b1218f01e9c [id=93178d6008234770bd8dcab5c797b068]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.903282 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b736bb7d13d4837af468a54208a4a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.896486 28468 tablet_service.cc:1467] Processing CreateTablet for tablet d064d7ffb2654b239e6695af35108cf0 (DEFAULT_TABLE table=default.Table_d67437cde20547fb8188102d8a8e9d54 [id=1c47e63a5a104968a994768fd2b67c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.904527 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d064d7ffb2654b239e6695af35108cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.908483 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a7e0f4297f4b638466c51798feed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.909505 30739 raft_consensus.cc:482] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.909662 30739 raft_consensus.cc:504] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.910343 30739 leader_election.cc:288] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.911221 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f080c95ff5ca41e0b3d27382040107e9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.911859 28150 leader_election.cc:341] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.912544 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f080c95ff5ca41e0b3d27382040107e9" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:32.913395 28150 leader_election.cc:341] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.913523 28150 leader_election.cc:302] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.913592 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 9897f05f81ef44e0ae272b11d7eaf1be (DEFAULT_TABLE table=default.Table_9099811cf7874977804669fff91540ca [id=9aee19c788de476f89647b5c515128a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.914008 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9897f05f81ef44e0ae272b11d7eaf1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.914512 30739 raft_consensus.cc:2726] T f080c95ff5ca41e0b3d27382040107e9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.922915 28600 tablet_bootstrap.cc:492] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:32.923089 28600 ts_tablet_manager.cc:1367] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.021s
I20240503 14:12:32.923828 28600 raft_consensus.cc:348] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.924063 28600 raft_consensus.cc:374] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.924183 28600 raft_consensus.cc:729] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.924444 28600 consensus_queue.cc:260] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.938644 28027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_6b41ac2dcacf495bbb627aa567c42764" } new_table_name: "default.Table_2cf90974d61146feb506d08c08ddd808" modify_external_catalogs: true
W20240503 14:12:32.939198 28026 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.194:43807 (request call id 117) took 2023 ms (client timeout 1000 ms). Trace:
W20240503 14:12:32.939298 28026 rpcz_store.cc:269] 0503 14:12:30.915906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.053464 (+137558us) service_pool.cc:225] Handling call
0503 14:12:32.939080 (+1885616us) spinlock_profiling.cc:243] Waited 82.4 ms on lock 0x60f00002ee78. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 000000000095519a 00007fe1198ea19d 00007fe1198e9dbe 00007fe12cda362d 00007fe12cda35cc 00007fe11aa62283 00007fe12cedc09b 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e
0503 14:12:32.939170 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:31.242960 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:31.243091 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:31.243099 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:31.243101 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:31.243230 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:31.243247 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:31.243252 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:31.243342 (+ 90us) tablet.cc:699] Row locks acquired
0503 14:12:31.243345 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:31.243391 (+ 46us) write_op.cc:270] Start()
0503 14:12:31.243431 (+ 40us) write_op.cc:276] Timestamp: P: 1714745551243353 usec, L: 0
0503 14:12:31.243435 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:31.243656 (+ 221us) log.cc:842] Serialized 356 byte log entry
0503 14:12:31.315827 (+ 72171us) op_driver.cc:444] REPLICATION: finished
0503 14:12:31.316281 (+ 454us) write_op.cc:301] APPLY: starting
0503 14:12:31.316306 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:31.316365 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:31.316398 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:31.316469 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:31.316508 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:31.316517 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:12:31.316604 (+ 87us) log.cc:842] Serialized 23 byte log entry
0503 14:12:31.316685 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:31.316708 (+ 23us) write_op.cc:454] Released schema lock
0503 14:12:31.316717 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:31.316781 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:31.316785 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0503 14:12:32.758297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:32.758468 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:32.758476 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:32.758479 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:32.758614 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:32.758630 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:32.758636 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:32.758718 (+ 82us) tablet.cc:699] Row locks acquired
0503 14:12:32.758721 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:32.758770 (+ 49us) write_op.cc:270] Start()
0503 14:12:32.758815 (+ 45us) write_op.cc:276] Timestamp: P: 1714745552758632 usec, L: 0
0503 14:12:32.758819 (+ 4us) op_driver.cc:331] REPLICATION: starting
0503 14:12:32.759077 (+ 258us) log.cc:842] Serialized 356 byte log entry
0503 14:12:32.788233 (+ 29156us) op_driver.cc:444] REPLICATION: finished
0503 14:12:32.790166 (+ 1933us) write_op.cc:301] APPLY: starting
0503 14:12:32.790176 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:32.790205 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:32.790226 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:32.790267 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:32.790292 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:32.790299 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:32.790354 (+ 55us) log.cc:842] Serialized 23 byte log entry
0503 14:12:32.790407 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.790420 (+ 13us) write_op.cc:454] Released schema lock
0503 14:12:32.790425 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:32.790470 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.790473 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0503 14:12:32.808798 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:32.808884 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:32.808890 (+ 6us) write_op.cc:435] Acquired schema lock
0503 14:12:32.808892 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:32.808965 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:32.808973 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:32.808978 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:32.809022 (+ 44us) tablet.cc:699] Row locks acquired
0503 14:12:32.809025 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:32.809056 (+ 31us) write_op.cc:270] Start()
0503 14:12:32.809084 (+ 28us) write_op.cc:276] Timestamp: P: 1714745552808919 usec, L: 0
0503 14:12:32.809087 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:12:32.809219 (+ 132us) log.cc:842] Serialized 356 byte log entry
0503 14:12:32.841521 (+ 32302us) op_driver.cc:444] REPLICATION: finished
0503 14:12:32.843692 (+ 2171us) write_op.cc:301] APPLY: starting
0503 14:12:32.843702 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:32.843730 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:32.843753 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:32.843809 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:32.843835 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:32.843849 (+ 14us) write_op.cc:312] APPLY: finished
0503 14:12:32.843895 (+ 46us) log.cc:842] Serialized 23 byte log entry
0503 14:12:32.843934 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.843945 (+ 11us) write_op.cc:454] Released schema lock
0503 14:12:32.843960 (+ 15us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:32.844009 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.844012 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7275,"reported_tablets":11,"spinlock_wait_cycles":189533440,"child_traces":[["op",{"apply.queue_time_us":388,"apply.run_cpu_time_us":566,"apply.run_wall_time_us":565,"num_ops":1,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":784,"prepare.run_wall_time_us":784,"replication_time_us":72356}],["op",{"apply.queue_time_us":1905,"apply.run_cpu_time_us":350,"apply.run_wall_time_us":349,"num_ops":1,"prepare.queue_time_us":9141,"prepare.run_cpu_time_us":888,"prepare.run_wall_time_us":890,"replication_time_us":29400}],["op",{"apply.queue_time_us":2142,"apply.run_cpu_time_us":360,"apply.run_wall_time_us":360,"num_ops":1,"prepare.queue_time_us":5536,"prepare.run_cpu_time_us":471,"prepare.run_wall_time_us":469,"replication_time_us":32422}]]}
I20240503 14:12:32.940821 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f760a9cbcc20443ba5c28ec36efb98db" } new_table_name: "default.Table_b3fa4232f73f40d28c53ef532d36d38b" modify_external_catalogs: true
W20240503 14:12:32.943374 28032 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.193:45997 (request call id 107) took 1723 ms (client timeout 1000 ms). Trace:
W20240503 14:12:32.943492 28032 rpcz_store.cc:269] 0503 14:12:31.219706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.232805 (+ 13099us) service_pool.cc:225] Handling call
0503 14:12:32.943051 (+1710246us) spinlock_profiling.cc:243] Waited 69.7 ms on lock 0x60f00002ee78. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 000000000095519a 00007fe1198ea19d 00007fe1198e9dbe 00007fe12cda362d 00007fe12cda35cc 00007fe11aa62283 00007fe12cedc09b 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e
0503 14:12:32.943342 (+ 291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:32.427150 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:32.427330 (+ 180us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:32.427340 (+ 10us) write_op.cc:435] Acquired schema lock
0503 14:12:32.427344 (+ 4us) tablet.cc:660] Decoding operations
0503 14:12:32.427485 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:32.427502 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:32.427508 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:32.427593 (+ 85us) tablet.cc:699] Row locks acquired
0503 14:12:32.427595 (+ 2us) write_op.cc:260] PREPARE: finished
0503 14:12:32.427643 (+ 48us) write_op.cc:270] Start()
0503 14:12:32.427687 (+ 44us) write_op.cc:276] Timestamp: P: 1714745552427498 usec, L: 0
0503 14:12:32.427692 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:32.427962 (+ 270us) log.cc:842] Serialized 356 byte log entry
0503 14:12:32.446053 (+ 18091us) op_driver.cc:444] REPLICATION: finished
0503 14:12:32.448306 (+ 2253us) write_op.cc:301] APPLY: starting
0503 14:12:32.448319 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:32.448349 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:32.448372 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:32.448408 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:32.448445 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:32.448452 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:12:32.448499 (+ 47us) log.cc:842] Serialized 23 byte log entry
0503 14:12:32.448535 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.448550 (+ 15us) write_op.cc:454] Released schema lock
0503 14:12:32.448556 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:32.448603 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.448607 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7161,"reported_tablets":53,"spinlock_wait_cycles":160236416,"child_traces":[["op",{"apply.queue_time_us":2216,"apply.run_cpu_time_us":340,"apply.run_wall_time_us":350,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":916,"prepare.run_wall_time_us":915,"replication_time_us":18346}]]}
I20240503 14:12:32.946408 28032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_07a11c032b6a448a8090c98f572b3dff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:32.956012 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 115061us
I20240503 14:12:32.956823 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc90d9b37bc4c57af3d29ce0f7da287 with delete_type TABLET_DATA_DELETED (replaced by tablet ba0e8d36b7bf4bdb8111673121e2bfbe) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.957854 28600 ts_tablet_manager.cc:1398] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.035s user 0.000s sys 0.002s
I20240503 14:12:32.957876 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc90d9b37bc4c57af3d29ce0f7da287 with delete_type TABLET_DATA_DELETED (replaced by tablet ba0e8d36b7bf4bdb8111673121e2bfbe) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.958593 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.959821 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet a31555fc800142bd849dbd572a194002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.960214 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.960433 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.960630 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.960801 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.960966 29828 tablet_replica.cc:330] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:32.961091 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7ca19dd15f48a68ace7fefc7ae660f with delete_type TABLET_DATA_DELETED (replaced by tablet 8ccb01614dee43e48d7e997bcfc7f6a6) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.961184 29828 raft_consensus.cc:2229] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.961242 28335 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c0540960ac45aabeaf13f888caf238) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.961373 29828 raft_consensus.cc:2258] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.961515 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.961717 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:21 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.961759 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet ac752d8ab4fd4a218b052f388f416494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.961920 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.961983 29828 ts_tablet_manager.cc:1875] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.962518 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cc84cf703048148abfe03b7dee12ce with delete_type TABLET_DATA_DELETED (replaced by tablet 5d308130f0dc4c16ac73058db332b41d) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.963081 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet a26e8389639748ae99832d7a7a9d5af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.963229 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 646f83185d5241268d4126d10561e937 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:32.963634 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b52ef7ecedf4273939406581d843980 with delete_type TABLET_DATA_DELETED (replaced by tablet 7bcfe8fff82a4fdd8c01a24a207a27ab) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.964226 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fe462c9ee54a27b9714f6457036d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.964273 30729 tablet_replica.cc:330] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:32.964524 30729 raft_consensus.cc:2229] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.964696 30729 raft_consensus.cc:2258] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.964792 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet ea261d8d5d174bee804aa2e171f35008 with delete_type TABLET_DATA_DELETED (replaced by tablet ae05e174ccbd4a588cf7ffb1de42632e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.965277 30729 ts_tablet_manager.cc:1875] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.965348 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fa067359a640eeba8c80d08371904e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:32.966856 28600 tablet_bootstrap.cc:492] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:32.967597 30739 raft_consensus.cc:482] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.967773 30739 raft_consensus.cc:504] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.968488 30739 leader_election.cc:288] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.969146 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d09bee02c14822aa08050eac5be181" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:32.969864 28150 leader_election.cc:341] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.970136 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d09bee02c14822aa08050eac5be181" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:32.970686 28150 leader_election.cc:341] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.970849 28150 leader_election.cc:302] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.971652 30739 raft_consensus.cc:2726] T 79d09bee02c14822aa08050eac5be181 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.972024 30739 raft_consensus.cc:482] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:32.972172 30739 raft_consensus.cc:504] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.972853 30739 leader_election.cc:288] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:32.973687 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c7b5fb9b4a483c8292f68387591034" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:32.974174 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2a3432263440f7a867dfb10249bc2d (DEFAULT_TABLE table=default.Table_1fbfaf1680ed48ef8c60061da7249ad9 [id=e8904211437b4d8c9d275cd69b2935d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240503 14:12:32.974438 28150 leader_election.cc:341] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.974653 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2a3432263440f7a867dfb10249bc2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:32.975039 28032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07a11c032b6a448a8090c98f572b3dff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:32.975976 30729 ts_tablet_manager.cc:1888] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:32.976469 28033 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 97) took 770 ms (client timeout 1000 ms). Trace:
W20240503 14:12:32.976941 28033 rpcz_store.cc:269] 0503 14:12:32.205955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:32.380011 (+174056us) service_pool.cc:225] Handling call
0503 14:12:32.936334 (+556323us) spinlock_profiling.cc:243] Waited 65.3 ms on lock 0x60f00002ee78. stack: 00007fe1198f4756 00007fe1198f4510 00007fe11892748d 0000000000955804 00000000009557cd 000000000095519a 00007fe1198ea19d 00007fe1198e9dbe 00007fe12cda362d 00007fe12cda35cc 00007fe11aa62283 00007fe12cedc09b 00007fe123ecd786 00007fe123ecd4b5 00007fe1205fc04e 00007fe1205faf2e
0503 14:12:32.976438 (+ 40104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:32.539727 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:32.539873 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:32.539880 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:12:32.539882 (+ 2us) tablet.cc:660] Decoding operations
0503 14:12:32.540012 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:32.540026 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:32.540031 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:32.540223 (+ 192us) tablet.cc:699] Row locks acquired
0503 14:12:32.540226 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:12:32.540277 (+ 51us) write_op.cc:270] Start()
0503 14:12:32.540318 (+ 41us) write_op.cc:276] Timestamp: P: 1714745552540123 usec, L: 0
0503 14:12:32.540323 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:32.540747 (+ 424us) log.cc:842] Serialized 533 byte log entry
0503 14:12:32.667234 (+126487us) op_driver.cc:444] REPLICATION: finished
0503 14:12:32.669012 (+ 1778us) write_op.cc:301] APPLY: starting
0503 14:12:32.669023 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:32.669052 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:32.669074 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:32.669111 (+ 37us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:32.669136 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:32.669142 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:12:32.669199 (+ 57us) log.cc:842] Serialized 23 byte log entry
0503 14:12:32.669257 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.669274 (+ 17us) write_op.cc:454] Released schema lock
0503 14:12:32.669279 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:32.669329 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.669332 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":35,"spinlock_wait_cycles":150277888,"child_traces":[["op",{"apply.queue_time_us":1748,"apply.run_cpu_time_us":363,"apply.run_wall_time_us":363,"num_ops":1,"prepare.queue_time_us":5508,"prepare.run_cpu_time_us":872,"prepare.run_wall_time_us":1638,"replication_time_us":126897}]]}
I20240503 14:12:32.977746 29828 ts_tablet_manager.cc:1888] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:32.977876 29828 log.cc:1196] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/a31555fc800142bd849dbd572a194002
I20240503 14:12:32.978188 29828 ts_tablet_manager.cc:1909] T a31555fc800142bd849dbd572a194002 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:32.974669 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c7b5fb9b4a483c8292f68387591034" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:32.978439 30729 log.cc:1196] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/7b7ca19dd15f48a68ace7fefc7ae660f
I20240503 14:12:32.978787 30729 ts_tablet_manager.cc:1909] T 7b7ca19dd15f48a68ace7fefc7ae660f P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
W20240503 14:12:32.979143 28150 leader_election.cc:341] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:32.979286 28150 leader_election.cc:302] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:32.979697 30739 raft_consensus.cc:2726] T d9c7b5fb9b4a483c8292f68387591034 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:32.979676 28464 tablet_service.cc:1467] Processing CreateTablet for tablet ca020cc71952417d821b9c7a6cb7b356 (DEFAULT_TABLE table=default.Table_ee4900123ee64b90bfc6b5e340346bcf [id=9e91bec22b8d4ce2ab424b720a33f3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.980201 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca020cc71952417d821b9c7a6cb7b356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.980724 30729 tablet_replica.cc:330] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:32.980991 29828 tablet_replica.cc:330] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:32.984792 29828 raft_consensus.cc:2229] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.985179 30729 raft_consensus.cc:2229] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.985539 30729 raft_consensus.cc:2258] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.985730 29828 pending_rounds.cc:62] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:32.985832 29828 pending_rounds.cc:68] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597777409978368 op_type: NO_OP noop_request { }
W20240503 14:12:32.986356 28025 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.27.50.195:55307 (request call id 96) took 1851 ms (client timeout 1000 ms). Trace:
I20240503 14:12:32.986506 28614 tablet_bootstrap.cc:492] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:32.986650 28614 ts_tablet_manager.cc:1367] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.007s
I20240503 14:12:32.986661 29828 raft_consensus.cc:2866] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
W20240503 14:12:32.986472 28025 rpcz_store.cc:269] 0503 14:12:31.134719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.201402 (+ 66683us) service_pool.cc:225] Handling call
0503 14:12:32.986330 (+1784928us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:12:31.879879 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0503 14:12:31.882287 (+ 2408us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:12:31.882295 (+ 8us) write_op.cc:435] Acquired schema lock
0503 14:12:31.882298 (+ 3us) tablet.cc:660] Decoding operations
0503 14:12:31.888136 (+ 5838us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:12:31.888160 (+ 24us) write_op.cc:641] Partition lock acquired for write op
0503 14:12:31.888180 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:12:31.889475 (+ 1295us) tablet.cc:699] Row locks acquired
0503 14:12:31.889485 (+ 10us) write_op.cc:260] PREPARE: finished
0503 14:12:31.889573 (+ 88us) write_op.cc:270] Start()
0503 14:12:31.889644 (+ 71us) write_op.cc:276] Timestamp: P: 1714745551889459 usec, L: 0
0503 14:12:31.889649 (+ 5us) op_driver.cc:331] REPLICATION: starting
0503 14:12:31.890733 (+ 1084us) log.cc:842] Serialized 533 byte log entry
0503 14:12:32.153190 (+262457us) op_driver.cc:444] REPLICATION: finished
0503 14:12:32.155304 (+ 2114us) write_op.cc:301] APPLY: starting
0503 14:12:32.155315 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0503 14:12:32.155339 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0503 14:12:32.155359 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:12:32.155397 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:12:32.155420 (+ 23us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0503 14:12:32.155425 (+ 5us) write_op.cc:312] APPLY: finished
0503 14:12:32.155473 (+ 48us) log.cc:842] Serialized 23 byte log entry
0503 14:12:32.155508 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.155521 (+ 13us) write_op.cc:454] Released schema lock
0503 14:12:32.155526 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0503 14:12:32.155570 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:12:32.155573 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":2086,"apply.run_cpu_time_us":306,"apply.run_wall_time_us":305,"num_ops":1,"prepare.queue_time_us":37757,"prepare.run_cpu_time_us":1472,"prepare.run_wall_time_us":31685,"replication_time_us":263438}]]}
I20240503 14:12:32.986951 29828 raft_consensus.cc:2258] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.987418 28614 raft_consensus.cc:348] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.987620 28614 raft_consensus.cc:374] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:32.987674 28614 raft_consensus.cc:729] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:32.987910 28614 consensus_queue.cc:260] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:32.988687 29828 ts_tablet_manager.cc:1875] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.988698 28456 tablet_service.cc:1514] Processing DeleteTablet for tablet a83532bf2852473a9c51ce7cc78b5d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:32.988924 30729 ts_tablet_manager.cc:1875] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:32.989406 28456 tablet_service.cc:1467] Processing CreateTablet for tablet 77d8ce2e91b74293a2652ab3d4bee82d (DEFAULT_TABLE table=default.Table_390a8ae260bb40809e52f71153fa46f0 [id=02a195b040b24f7981ad3d6f42edb9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.989859 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d8ce2e91b74293a2652ab3d4bee82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:32.991180 28600 tablet_bootstrap.cc:654] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:32.994544 28462 tablet_service.cc:1467] Processing CreateTablet for tablet ee0897770c0a4a38b6a77624f630f722 (DEFAULT_TABLE table=default.Table_02b32574c2ac4fdab08b851ecd21084e [id=2ce3f18a2b1443bdb94dac7c6b44abc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:32.994989 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0897770c0a4a38b6a77624f630f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.002557 29828 ts_tablet_manager.cc:1888] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.012750 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 137c9eef465843049e93374367e562af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.013259 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8e6e922f5b4577861af5bdbee94f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.013614 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet a05f786073084bd1b8c472c20dc8bff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b828eb16891434f9031b9e4d29b18d8) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.013911 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d18fdc987d4879a642438e748eeb62 with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24f19b95e44f48f76fea5a22ba1c1) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.014241 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc90d9b37bc4c57af3d29ce0f7da287 with delete_type TABLET_DATA_DELETED (replaced by tablet ba0e8d36b7bf4bdb8111673121e2bfbe) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.014485 29828 log.cc:1196] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/5cdc868ddf0a490292eae78ebc8c8ec6
I20240503 14:12:33.014552 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.014856 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.015206 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 6696a237c3dd428097e2ebdf24afd63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.016690 30729 ts_tablet_manager.cc:1888] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.015465 29828 ts_tablet_manager.cc:1909] T 5cdc868ddf0a490292eae78ebc8c8ec6 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.017526 30729 log.cc:1196] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/9dcf8bae3b5e4ff69a2afedb6c466ae4
I20240503 14:12:33.017508 28471 tablet_service.cc:1514] Processing DeleteTablet for tablet 34238c9859724d97a32f2a1d9315edb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e910e098ec44a07af6ed8ff4647fc7e) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.017994 30729 ts_tablet_manager.cc:1909] T 9dcf8bae3b5e4ff69a2afedb6c466ae4 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:33.018177 28462 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de9e159d2ba47039dd6bd207a9cddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.018370 29828 tablet_replica.cc:330] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.018630 29828 raft_consensus.cc:2229] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.019821 30729 tablet_replica.cc:330] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:33.020099 29828 raft_consensus.cc:2258] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.020360 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996bb3a7edfc4f5a8a3a39ea6fcfe5cd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.020612 28343 raft_consensus.cc:2454] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:33.020800 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet b59945d011b64cba87c854e2ffde8226 with delete_type TABLET_DATA_DELETED (replaced by tablet 47965a978a904f2d876690de01f89771) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.021068 29828 ts_tablet_manager.cc:1875] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.021406 30729 raft_consensus.cc:2229] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.021689 30729 raft_consensus.cc:2258] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.022419 30729 ts_tablet_manager.cc:1875] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.024276 28614 ts_tablet_manager.cc:1398] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20240503 14:12:33.034428 28614 tablet_bootstrap.cc:492] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:33.039391 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00dbc5a90674873b61b377e3e8d92fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:33.039965 29828 ts_tablet_manager.cc:1888] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.040124 29828 log.cc:1196] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7d348da459634561985c4ac1710c0bb2
I20240503 14:12:33.040406 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00dbc5a90674873b61b377e3e8d92fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:33.040602 28212 raft_consensus.cc:3037] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.041505 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c79ffee3954b5c870f4a8aa6379517 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c0540960ac45aabeaf13f888caf238) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.041792 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet ac752d8ab4fd4a218b052f388f416494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.042060 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet f37ea39f88b04f0abb0445d4342406c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.043287 28212 raft_consensus.cc:2454] T c00dbc5a90674873b61b377e3e8d92fb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:33.044924 28179 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ebec4622ef4950bd1c70451958c59b with delete_type TABLET_DATA_DELETED (replaced by tablet fc3f7c221cbc467dabfe5665ccd07d48) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.044976 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet e69e01bdbae349d0a61a295b6f6352f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 29c78b82bd264b53ba60583e6d8f06d4) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.045495 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d09bee02c14822aa08050eac5be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.045950 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86451bc448428da7485e70e0d54b89 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cad03adb3ee419c962aaf20c72f8c04) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.046329 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet e45df464f6954a56bec8b107b350320c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.046754 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f75e102dd4249c3800e1fcb4339acf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.047142 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3ce05b422a4166bfac468acb84e4cc with delete_type TABLET_DATA_DELETED (replaced by tablet 64f226db2fdb47ba9472a3aa65b953c3) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.047274 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a05f786073084bd1b8c472c20dc8bff6 (table default.Table_f010ec576f0a40cb91c71b23da613670 [id=3221ffb751294f5b9f7fa7848d9845e3]) successfully deleted
W20240503 14:12:33.047564 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f7d18fdc987d4879a642438e748eeb62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7d18fdc987d4879a642438e748eeb62 already in progress: creating tablet
I20240503 14:12:33.047559 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 007c84fd9d9247728b0babd48689aba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4451a8c3ab1e4fccac7100ac74bb990f) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:33.047783 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet bbc90d9b37bc4c57af3d29ce0f7da287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbc90d9b37bc4c57af3d29ce0f7da287 already in progress: creating tablet
W20240503 14:12:33.047997 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a035a5249f8348b591b27ba3fa7ae43a because the tablet was not found. No further retry: Not found: Tablet not found: a035a5249f8348b591b27ba3fa7ae43a
I20240503 14:12:33.048269 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.048624 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a58db50f849a8be729c10dbe03e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:18 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.048985 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b94092725fa4956a4c00e3685f9aeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.049345 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c8a97076a44f9eaa3a15026f90a304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:12 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.049690 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.050065 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 34238c9859724d97a32f2a1d9315edb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e910e098ec44a07af6ed8ff4647fc7e) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.050045 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet a035a5249f8348b591b27ba3fa7ae43a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.050433 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.050781 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7779f7a9544e17aea7135eab2ccd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.051155 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet aac51b83abe14ab5b1fb5d627233e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.051558 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.052017 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet cf810e84367f420b928a171790849445 with delete_type TABLET_DATA_DELETED (replaced by tablet f154dfaeb4fe47238ad928c8fc3521da) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.057041 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet a31555fc800142bd849dbd572a194002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:19 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.057564 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet ac752d8ab4fd4a218b052f388f416494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.063431 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f5103b31ed45b480252dcf94b72f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.064491 28661 ts_tablet_manager.cc:1875] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.066244 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet e220336186f648419b186feada6f0124 with delete_type TABLET_DATA_DELETED (replaced by tablet a83532bf2852473a9c51ce7cc78b5d34) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.067373 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet b59945d011b64cba87c854e2ffde8226 with delete_type TABLET_DATA_DELETED (replaced by tablet 47965a978a904f2d876690de01f89771) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.067971 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.068758 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d348da459634561985c4ac1710c0bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:26 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.069200 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd6d40a57e4bea973874ff12c0ea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:27 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.080726 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebec4622ef4950bd1c70451958c59b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:33.081179 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet a31555fc800142bd849dbd572a194002 (table default.Table_d7b3acfa64194593899187c496e64c66 [id=5d99991da3ec4a0e9ea84e4bb6762014]) successfully deleted
I20240503 14:12:33.081537 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 5cdc868ddf0a490292eae78ebc8c8ec6 (table default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3 [id=ca041900898046468ff2eec0d0fadd85]) successfully deleted
I20240503 14:12:33.081843 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a05f786073084bd1b8c472c20dc8bff6 (table default.Table_f010ec576f0a40cb91c71b23da613670 [id=3221ffb751294f5b9f7fa7848d9845e3]) successfully deleted
W20240503 14:12:33.082139 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet f7d18fdc987d4879a642438e748eeb62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7d18fdc987d4879a642438e748eeb62 already in progress: creating tablet
W20240503 14:12:33.082407 28017 catalog_manager.cc:4810] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): delete failed for tablet bbc90d9b37bc4c57af3d29ce0f7da287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbc90d9b37bc4c57af3d29ce0f7da287 already in progress: creating tablet
I20240503 14:12:33.082698 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 7b7ca19dd15f48a68ace7fefc7ae660f (table default.Table_d6bd120570ec4a8ab95342666e7d1d83 [id=84f355c001064d28a408a8012a24df78]) successfully deleted
I20240503 14:12:33.082942 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 9dcf8bae3b5e4ff69a2afedb6c466ae4 (table default.Table_04ae17bd98c04c4b993d808882cdc3ec [id=5629316959054b54a63bac11c6d3d8d6]) successfully deleted
I20240503 14:12:33.086154 28030 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9dc8c5fcf27e4fddac5ff39852a75d20" } modify_external_catalogs: true
I20240503 14:12:33.086355 28030 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_9dc8c5fcf27e4fddac5ff39852a75d20" } modify_external_catalogs: true
I20240503 14:12:33.087225 28028 master_service.cc:946] ReplaceTablet: received request to replace tablet 6a9f6a750b744433b01f821506767a02 from {username='slave'} at 127.0.0.1:33344
I20240503 14:12:33.088526 28029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ac11e443561a400ea0c013dd4e9178cf" } modify_external_catalogs: true
I20240503 14:12:33.088721 28029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_ac11e443561a400ea0c013dd4e9178cf" } modify_external_catalogs: true
I20240503 14:12:33.094106 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 0c7779f7a9544e17aea7135eab2ccd5f (table default.Table_2a170feff0884500b519684ac43a06b0 [id=2d1a26ea051048589afec842f2a31c72]) successfully deleted
I20240503 14:12:33.095360 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 5cdc868ddf0a490292eae78ebc8c8ec6 (table default.Table_2aeb5930b9a34c4b92b8ae398b0bc5a3 [id=ca041900898046468ff2eec0d0fadd85]) successfully deleted
I20240503 14:12:33.096870 28604 tablet_bootstrap.cc:492] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:33.097128 28604 ts_tablet_manager.cc:1367] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.328s user 0.012s sys 0.008s
I20240503 14:12:33.098268 28604 raft_consensus.cc:348] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.098719 28604 raft_consensus.cc:374] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.098835 28604 raft_consensus.cc:729] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.099155 28604 consensus_queue.cc:260] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.099853 28604 ts_tablet_manager.cc:1398] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:33.100394 28604 tablet_bootstrap.cc:492] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:33.102982 28661 ts_tablet_manager.cc:1888] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.104976 28343 raft_consensus.cc:3037] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.110280 29828 ts_tablet_manager.cc:1909] T 7d348da459634561985c4ac1710c0bb2 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.110862 29828 tablet_replica.cc:330] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.111069 29828 raft_consensus.cc:2229] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.111248 29828 raft_consensus.cc:2258] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.111460 28032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5f8d6960d51e4a63a42f434099952825" } modify_external_catalogs: true
I20240503 14:12:33.111668 28032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_5f8d6960d51e4a63a42f434099952825" } modify_external_catalogs: true
W20240503 14:12:33.112192 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 353d55280fac4953a54347c3756adcee consensus state has no leader
I20240503 14:12:33.112545 28026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2aadd33c71aa4953a09c3d7a747e65da" } modify_external_catalogs: true
I20240503 14:12:33.112699 28026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_2aadd33c71aa4953a09c3d7a747e65da" } modify_external_catalogs: true
I20240503 14:12:33.112630 30759 tablet.cc:1722] T 780b53efefcf42569bba2fcabd21e211 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240503 14:12:33.113723 28027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 812d20a15f3b4fa1ad43851b7aea56bc consensus state has no leader
I20240503 14:12:33.113842 30729 ts_tablet_manager.cc:1888] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.113950 30729 log.cc:1196] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/08cc84cf703048148abfe03b7dee12ce
I20240503 14:12:33.114318 30729 ts_tablet_manager.cc:1909] T 08cc84cf703048148abfe03b7dee12ce P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:33.114143 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_bbc6826438b64a6ba3c3102183e64721"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:33.114720 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc6826438b64a6ba3c3102183e64721 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.117666 28031 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 888636e9beca43edab96a8736751409e (table default.Table_a13cfbd07ce348578e9883c1205e8b9b [id=3854b08f7cd348c897aac4aacb7f5d06]). Expected version 3 got 0
I20240503 14:12:33.117978 28031 catalog_manager.cc:5576] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_ec97037ea5a849bfb48a72d6771fbf1f [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 2 got 0
I20240503 14:12:33.119990 29828 ts_tablet_manager.cc:1875] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.121305 28025 catalog_manager.cc:5484] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:33.121765 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 888636e9beca43edab96a8736751409e (table default.Table_a13cfbd07ce348578e9883c1205e8b9b [id=3854b08f7cd348c897aac4aacb7f5d06]). Expected version 3 got 0
I20240503 14:12:33.122108 28025 catalog_manager.cc:5484] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc reported cstate change: term changed from 0 to 1, leader changed from <none> to d472d9f94d8949cf86284d32a161cebc (127.27.50.195). New cstate: current_term: 1 leader_uuid: "d472d9f94d8949cf86284d32a161cebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:33.122650 28025 catalog_manager.cc:5576] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233) does not have the latest schema for tablet 2f7b805aee554f8fa4c198ce3409c6e6 (table default.Table_ee9f2d2bd2b54111963a23deaf110216 [id=4df57b62a5b34ac182e799495a848041]). Expected version 1 got 0
I20240503 14:12:33.127939 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_ec97037ea5a849bfb48a72d6771fbf1f [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 2 got 0
I20240503 14:12:33.128255 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 5bf863dabf6647618732a03af84efab3 (table default.Table_3c0e46c1611c4132ae4490057c568592 [id=e47d9e4b4933422e962efd3117aea50e]). Expected version 2 got 0
I20240503 14:12:33.128484 28033 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 218481d1ddd246cf9aa9c0e163666f4e (table default.Table_7d54afe4afa34f16bd9e0d4e0c61ab72 [id=4bfb3a5cb63f4af2a4abbab25f2306b5]). Expected version 2 got 0
I20240503 14:12:33.138489 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3c30b4b6a28a49349e0ed98bf004330a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:33.144632 28614 tablet_bootstrap.cc:654] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:33.145265 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c30b4b6a28a49349e0ed98bf004330a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.146857 28031 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_bdce2849aeb843c0a01b36cd7dbcaee6" } new_table_name: "default.Table_edfb9c2f1c36490c985b359f55c50235" modify_external_catalogs: true
I20240503 14:12:33.151345 30616 raft_consensus.cc:482] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.151531 30616 raft_consensus.cc:504] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.152248 30616 leader_election.cc:288] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.161729 29828 ts_tablet_manager.cc:1888] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.166478 29828 log.cc:1196] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/31ddd791d4e14630bfb3b4b1513f76e1
I20240503 14:12:33.167757 29828 ts_tablet_manager.cc:1909] T 31ddd791d4e14630bfb3b4b1513f76e1 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
W20240503 14:12:33.169116 28017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.253:44259, user_credentials={real_user=slave}} blocked reactor thread for 60317.6us
I20240503 14:12:33.170223 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 7a379840da72499e8e1413db59a76471 (DEFAULT_TABLE table=default.Table_76e967061ef74740937418c5be5d08a6 [id=44a6dfd7e096447484a64c80543b0a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.170606 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 152069b784d34a14975fcefb63a3a5c7 (DEFAULT_TABLE table=default.Table_84e92c206ac54e07845732e6239ab59a [id=995ce85508f6446eb30cce3f7d9b8fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.171010 28333 tablet_service.cc:1467] Processing CreateTablet for tablet c0f2fffa8fea4d928f1be502a270befb (DEFAULT_TABLE table=default.Table_7757fd0ba3ea4c5c85b42c1e3e3facc2 [id=db6ad41f0b464268b9616a26d5cf745b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.171607 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee37b4af78f4cedb00789dfe6ccedbb (DEFAULT_TABLE table=default.Table_5e84413c67034dc3ac76040484753c93 [id=b674ca9de63d4b0b88ab623051eed3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.172307 28335 tablet_service.cc:1467] Processing CreateTablet for tablet 783a1dc985ac447b8865e6c8b1c4fc58 (DEFAULT_TABLE table=default.Table_c184e04cc722407c84daac50b15af9e3 [id=db698185af714b9284334a4c2ba03777]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.176162 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 3269b413c6fb494187828380bc9b4788 (DEFAULT_TABLE table=default.Table_4d04603a0852462f8671107845e91561 [id=ddaf1b7c34b24b799318b5c90e44ea10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.176407 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b86451bc448428da7485e70e0d54b89 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cad03adb3ee419c962aaf20c72f8c04) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.172855 28337 tablet_service.cc:1467] Processing CreateTablet for tablet 3db4e380ecb044ebad1489e2c8d4a12e (DEFAULT_TABLE table=default.Table_3e8ffe332027447da16c340fb4fb6f5d [id=49d7fc343ed5469588be118c55d2587c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.173353 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet a05f786073084bd1b8c472c20dc8bff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b828eb16891434f9031b9e4d29b18d8) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.177222 28328 tablet_service.cc:1467] Processing CreateTablet for tablet 6a55c0ea2821423e9148e0ea07a04253 (DEFAULT_TABLE table=default.Table_6e7390ee45af41cb942b080a7f37b139 [id=74a55802c27b48f5a93e6ad1cb3da5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.177537 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 5cad03adb3ee419c962aaf20c72f8c04 (DEFAULT_TABLE table=default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.177868 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.179287 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3269b413c6fb494187828380bc9b4788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.179512 28333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f2fffa8fea4d928f1be502a270befb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.179878 29828 tablet_replica.cc:330] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.180066 29828 raft_consensus.cc:2229] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:33.180351 29828 pending_rounds.cc:62] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:33.180390 29828 pending_rounds.cc:68] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597782952624128 op_type: NO_OP noop_request { }
I20240503 14:12:33.180486 29828 raft_consensus.cc:2866] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:33.180557 29828 raft_consensus.cc:2258] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:33.182977 28290 leader_election.cc:341] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: aac51b83abe14ab5b1fb5d627233e58e
I20240503 14:12:33.183207 29828 ts_tablet_manager.cc:1875] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.183306 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebec4622ef4950bd1c70451958c59b" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:33.183507 28209 raft_consensus.cc:3037] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.183645 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a55c0ea2821423e9148e0ea07a04253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.184474 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783a1dc985ac447b8865e6c8b1c4fc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.184598 30739 raft_consensus.cc:482] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.185317 28180 tablet_service.cc:1467] Processing CreateTablet for tablet 7a379840da72499e8e1413db59a76471 (DEFAULT_TABLE table=default.Table_76e967061ef74740937418c5be5d08a6 [id=44a6dfd7e096447484a64c80543b0a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.188230 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee37b4af78f4cedb00789dfe6ccedbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.192181 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db4e380ecb044ebad1489e2c8d4a12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.193464 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 152069b784d34a14975fcefb63a3a5c7 (DEFAULT_TABLE table=default.Table_84e92c206ac54e07845732e6239ab59a [id=995ce85508f6446eb30cce3f7d9b8fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.184722 30739 raft_consensus.cc:504] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.195698 28196 tablet_service.cc:1467] Processing CreateTablet for tablet c0f2fffa8fea4d928f1be502a270befb (DEFAULT_TABLE table=default.Table_7757fd0ba3ea4c5c85b42c1e3e3facc2 [id=db6ad41f0b464268b9616a26d5cf745b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.195822 30739 leader_election.cc:288] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:33.196195 28192 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee37b4af78f4cedb00789dfe6ccedbb (DEFAULT_TABLE table=default.Table_5e84413c67034dc3ac76040484753c93 [id=b674ca9de63d4b0b88ab623051eed3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.196767 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 152069b784d34a14975fcefb63a3a5c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.197206 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf863dabf6647618732a03af84efab3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:33.198802 28150 leader_election.cc:341] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.201635 28209 raft_consensus.cc:2454] T a1ebec4622ef4950bd1c70451958c59b P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:33.203135 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a379840da72499e8e1413db59a76471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.207286 29828 ts_tablet_manager.cc:1888] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.207437 29828 log.cc:1196] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/d7c79ffee3954b5c870f4a8aa6379517
I20240503 14:12:33.207866 29828 ts_tablet_manager.cc:1909] T d7c79ffee3954b5c870f4a8aa6379517 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.209206 28180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a379840da72499e8e1413db59a76471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.210211 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 152069b784d34a14975fcefb63a3a5c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.211117 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f2fffa8fea4d928f1be502a270befb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.211514 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cad03adb3ee419c962aaf20c72f8c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.220819 29828 tablet_replica.cc:330] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.221522 29828 raft_consensus.cc:2229] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.221776 29828 raft_consensus.cc:2258] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.222486 29828 ts_tablet_manager.cc:1875] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.223166 28192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee37b4af78f4cedb00789dfe6ccedbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.108547 30733 consensus_queue.cc:1035] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:33.228803 30729 tablet_replica.cc:330] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:33.229074 30729 raft_consensus.cc:2229] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:33.109869 28427 consensus_peers.cc:485] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 989ea2f962a7492d8c32e4becec1457d. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:33.229342 28427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 119507us
W20240503 14:12:33.229614 28427 leader_election.cc:341] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.229744 28427 leader_election.cc:302] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c, d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:33.230108 30733 raft_consensus.cc:2726] T d6f5103b31ed45b480252dcf94b72f9c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:33.230264 28427 leader_election.cc:341] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.230871 28427 leader_election.cc:302] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:33.231101 30558 raft_consensus.cc:2781] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:33.231174 30558 raft_consensus.cc:482] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:12:33.231186 28427 leader_election.cc:341] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.231228 30558 raft_consensus.cc:3037] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:33.231397 28427 leader_election.cc:341] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.111769 30719 raft_consensus.cc:482] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.231860 30719 raft_consensus.cc:504] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.232501 30719 leader_election.cc:288] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.122462 28604 tablet_bootstrap.cc:654] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.153728 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac51b83abe14ab5b1fb5d627233e58e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:33.233444 28488 raft_consensus.cc:2454] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:33.196861 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf863dabf6647618732a03af84efab3" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:33.233964 30558 raft_consensus.cc:504] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.234472 28290 leader_election.cc:302] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:33.234632 30558 leader_election.cc:288] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.234936 30616 raft_consensus.cc:2781] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:12:33.234899 28150 leader_election.cc:341] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.235018 30616 raft_consensus.cc:482] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:33.235078 30616 raft_consensus.cc:3037] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.235054 28150 leader_election.cc:302] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:33.235529 30729 raft_consensus.cc:2258] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.236150 30729 ts_tablet_manager.cc:1875] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.236490 30739 raft_consensus.cc:2726] T 5bf863dabf6647618732a03af84efab3 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:33.237313 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 3269b413c6fb494187828380bc9b4788 (DEFAULT_TABLE table=default.Table_4d04603a0852462f8671107845e91561 [id=ddaf1b7c34b24b799318b5c90e44ea10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.237717 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3269b413c6fb494187828380bc9b4788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.237682 30616 raft_consensus.cc:504] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.238404 30616 leader_election.cc:288] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Requested vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.239498 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac51b83abe14ab5b1fb5d627233e58e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc"
I20240503 14:12:33.239739 28494 raft_consensus.cc:3037] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.240000 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 3db4e380ecb044ebad1489e2c8d4a12e (DEFAULT_TABLE table=default.Table_3e8ffe332027447da16c340fb4fb6f5d [id=49d7fc343ed5469588be118c55d2587c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.240429 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db4e380ecb044ebad1489e2c8d4a12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.240000 28188 tablet_service.cc:1467] Processing CreateTablet for tablet 783a1dc985ac447b8865e6c8b1c4fc58 (DEFAULT_TABLE table=default.Table_c184e04cc722407c84daac50b15af9e3 [id=db698185af714b9284334a4c2ba03777]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.241423 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783a1dc985ac447b8865e6c8b1c4fc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.242599 28494 raft_consensus.cc:2454] T aac51b83abe14ab5b1fb5d627233e58e P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 1.
I20240503 14:12:33.242784 30733 consensus_queue.cc:1035] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:33.243034 28187 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c17cfce7e644668817bf3483a098d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 10876f4512794e6aa72bda3c1da8b2d0) from {username='slave'} at 127.0.0.1:58172
W20240503 14:12:33.243355 28290 leader_election.cc:341] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: aac51b83abe14ab5b1fb5d627233e58e
I20240503 14:12:33.243690 28181 tablet_service.cc:1467] Processing CreateTablet for tablet 5cad03adb3ee419c962aaf20c72f8c04 (DEFAULT_TABLE table=default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.244151 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cad03adb3ee419c962aaf20c72f8c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.243690 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 6a55c0ea2821423e9148e0ea07a04253 (DEFAULT_TABLE table=default.Table_6e7390ee45af41cb942b080a7f37b139 [id=74a55802c27b48f5a93e6ad1cb3da5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.249292 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a55c0ea2821423e9148e0ea07a04253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.251672 28353 raft_consensus.cc:3037] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.257875 28290 leader_election.cc:302] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters: a64bc15d65d84ad5a005960a9a84df9c
I20240503 14:12:33.257814 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996bb3a7edfc4f5a8a3a39ea6fcfe5cd" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:33.258473 28354 raft_consensus.cc:3037] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.260521 30719 raft_consensus.cc:482] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.260676 30719 raft_consensus.cc:504] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.261283 30719 leader_election.cc:288] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.261339 28354 raft_consensus.cc:2454] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:33.261812 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fe462c9ee54a27b9714f6457036d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.261775 28343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7b805aee554f8fa4c198ce3409c6e6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.262071 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.262288 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ebec4622ef4950bd1c70451958c59b with delete_type TABLET_DATA_DELETED (replaced by tablet fc3f7c221cbc467dabfe5665ccd07d48) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.262465 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b21cd1262d94d1c994b29c44f86cb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:13 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.262661 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet aac51b83abe14ab5b1fb5d627233e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.262894 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet e220336186f648419b186feada6f0124 with delete_type TABLET_DATA_DELETED (replaced by tablet a83532bf2852473a9c51ce7cc78b5d34) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.264889 28457 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee37b4af78f4cedb00789dfe6ccedbb (DEFAULT_TABLE table=default.Table_5e84413c67034dc3ac76040484753c93 [id=b674ca9de63d4b0b88ab623051eed3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.265324 28457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee37b4af78f4cedb00789dfe6ccedbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.269209 28464 tablet_service.cc:1467] Processing CreateTablet for tablet 3db4e380ecb044ebad1489e2c8d4a12e (DEFAULT_TABLE table=default.Table_3e8ffe332027447da16c340fb4fb6f5d [id=49d7fc343ed5469588be118c55d2587c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.269696 28464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db4e380ecb044ebad1489e2c8d4a12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.272459 28472 tablet_service.cc:1467] Processing CreateTablet for tablet 3269b413c6fb494187828380bc9b4788 (DEFAULT_TABLE table=default.Table_4d04603a0852462f8671107845e91561 [id=ddaf1b7c34b24b799318b5c90e44ea10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.272984 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3269b413c6fb494187828380bc9b4788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.276513 28469 tablet_service.cc:1467] Processing CreateTablet for tablet 6a55c0ea2821423e9148e0ea07a04253 (DEFAULT_TABLE table=default.Table_6e7390ee45af41cb942b080a7f37b139 [id=74a55802c27b48f5a93e6ad1cb3da5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.277040 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a55c0ea2821423e9148e0ea07a04253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.278110 30729 ts_tablet_manager.cc:1888] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.278231 30729 log.cc:1196] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/a26e8389639748ae99832d7a7a9d5af3
I20240503 14:12:33.279019 30729 ts_tablet_manager.cc:1909] T a26e8389639748ae99832d7a7a9d5af3 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:33.280485 28474 tablet_service.cc:1467] Processing CreateTablet for tablet 5cad03adb3ee419c962aaf20c72f8c04 (DEFAULT_TABLE table=default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.281036 28474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cad03adb3ee419c962aaf20c72f8c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.282331 30616 raft_consensus.cc:482] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d472d9f94d8949cf86284d32a161cebc)
I20240503 14:12:33.282493 30616 raft_consensus.cc:504] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.283593 30616 leader_election.cc:288] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.284006 29828 ts_tablet_manager.cc:1888] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.284147 29828 log.cc:1196] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/ac752d8ab4fd4a218b052f388f416494
I20240503 14:12:33.284641 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 152069b784d34a14975fcefb63a3a5c7 (DEFAULT_TABLE table=default.Table_84e92c206ac54e07845732e6239ab59a [id=995ce85508f6446eb30cce3f7d9b8fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.284747 29828 ts_tablet_manager.cc:1909] T ac752d8ab4fd4a218b052f388f416494 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.285331 29828 tablet_replica.cc:330] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.285243 28466 tablet_service.cc:1467] Processing CreateTablet for tablet c0f2fffa8fea4d928f1be502a270befb (DEFAULT_TABLE table=default.Table_7757fd0ba3ea4c5c85b42c1e3e3facc2 [id=db6ad41f0b464268b9616a26d5cf745b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.285697 28459 tablet_service.cc:1467] Processing CreateTablet for tablet 7a379840da72499e8e1413db59a76471 (DEFAULT_TABLE table=default.Table_76e967061ef74740937418c5be5d08a6 [id=44a6dfd7e096447484a64c80543b0a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.286270 28459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a379840da72499e8e1413db59a76471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.288072 30616 raft_consensus.cc:2781] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:33.288334 30616 raft_consensus.cc:686] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Becoming Leader. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Running, Role: LEADER
I20240503 14:12:33.288421 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f2fffa8fea4d928f1be502a270befb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.288687 30616 consensus_queue.cc:237] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.292994 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 152069b784d34a14975fcefb63a3a5c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.296515 29828 raft_consensus.cc:2229] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.296981 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 783a1dc985ac447b8865e6c8b1c4fc58 (DEFAULT_TABLE table=default.Table_c184e04cc722407c84daac50b15af9e3 [id=db698185af714b9284334a4c2ba03777]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:33.297550 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783a1dc985ac447b8865e6c8b1c4fc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:33.298452 29828 raft_consensus.cc:2258] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.229624 28661 log.cc:1196] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/31ddd791d4e14630bfb3b4b1513f76e1
I20240503 14:12:33.299566 29828 ts_tablet_manager.cc:1875] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.299702 28661 ts_tablet_manager.cc:1909] T 31ddd791d4e14630bfb3b4b1513f76e1 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
I20240503 14:12:33.300616 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1163) took 2182 ms. Trace:
I20240503 14:12:33.300751 28661 rpcz_store.cc:276] 0503 14:12:31.118284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.118328 (+ 44us) service_pool.cc:225] Handling call
0503 14:12:32.862503 (+1744175us) ts_tablet_manager.cc:1120] Deleting tablet 31ddd791d4e14630bfb3b4b1513f76e1
0503 14:12:32.862514 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.068103 (+205589us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:33.102857 (+ 34754us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:33.300589 (+197732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1743897}
I20240503 14:12:33.301189 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1165) took 1970 ms. Trace:
I20240503 14:12:33.301294 28661 rpcz_store.cc:276] 0503 14:12:31.331005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.331303 (+ 298us) service_pool.cc:225] Handling call
0503 14:12:33.301055 (+1969752us) ts_tablet_manager.cc:1120] Deleting tablet 01482b478dcd45598fa65d14904fe276
0503 14:12:33.301065 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.301173 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969473}
I20240503 14:12:33.301667 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1166) took 1970 ms. Trace:
I20240503 14:12:33.301771 28661 rpcz_store.cc:276] 0503 14:12:31.331084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.331631 (+ 547us) service_pool.cc:225] Handling call
0503 14:12:33.301550 (+1969919us) ts_tablet_manager.cc:1120] Deleting tablet 61da48859c4d4a73bb1b9d7c0ede8106
0503 14:12:33.301559 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.301651 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969749}
I20240503 14:12:33.302085 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1167) took 1970 ms. Trace:
I20240503 14:12:33.302166 28661 rpcz_store.cc:276] 0503 14:12:31.331161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.331829 (+ 668us) service_pool.cc:225] Handling call
0503 14:12:33.301996 (+1970167us) ts_tablet_manager.cc:1120] Deleting tablet a88cf5de223f4a6abc21691fea1c4da0
0503 14:12:33.302005 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.302074 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970028}
I20240503 14:12:33.302551 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1169) took 1892 ms. Trace:
W20240503 14:12:33.303066 28427 consensus_peers.cc:485] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac752d8ab4fd4a218b052f388f416494. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:33.304695 28427 leader_election.cc:341] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 996bb3a7edfc4f5a8a3a39ea6fcfe5cd
W20240503 14:12:33.305246 28427 consensus_peers.cc:485] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b86451bc448428da7485e70e0d54b89. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:33.305557 28427 leader_election.cc:302] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:33.305833 30719 consensus_queue.cc:1035] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:33.306346 30719 raft_consensus.cc:2781] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:33.306470 30719 raft_consensus.cc:686] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:33.306752 30719 consensus_queue.cc:237] T c00dbc5a90674873b61b377e3e8d92fb P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.324801 29828 ts_tablet_manager.cc:1888] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.325034 29828 log.cc:1196] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/6696a237c3dd428097e2ebdf24afd63f
I20240503 14:12:33.325465 29828 ts_tablet_manager.cc:1909] T 6696a237c3dd428097e2ebdf24afd63f P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.326073 29828 tablet_replica.cc:330] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.328835 28604 tablet_bootstrap.cc:492] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:33.329114 28604 ts_tablet_manager.cc:1367] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.229s user 0.009s sys 0.005s
I20240503 14:12:33.330111 28604 raft_consensus.cc:348] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.330494 28604 raft_consensus.cc:374] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.330680 28604 raft_consensus.cc:729] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.331022 28604 consensus_queue.cc:260] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.303392 28661 rpcz_store.cc:276] 0503 14:12:31.409810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.409888 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:33.302456 (+1892568us) ts_tablet_manager.cc:1120] Deleting tablet 530f24322a454f7bb04e818ca865a369
0503 14:12:33.302465 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.302535 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882599}
I20240503 14:12:33.331879 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1170) took 1921 ms. Trace:
I20240503 14:12:33.331993 28604 ts_tablet_manager.cc:1398] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20240503 14:12:33.334626 28427 consensus_peers.cc:485] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a05f786073084bd1b8c472c20dc8bff6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:33.335466 28604 tablet_bootstrap.cc:492] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:33.332037 28661 rpcz_store.cc:276] 0503 14:12:31.409912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.410107 (+ 195us) service_pool.cc:225] Handling call
0503 14:12:33.331696 (+1921589us) ts_tablet_manager.cc:1120] Deleting tablet f783ee62adba49579a0518b240b795bc
0503 14:12:33.331710 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.331855 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911746}
I20240503 14:12:33.337633 30729 tablet_replica.cc:330] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:33.343186 29828 raft_consensus.cc:2229] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.343879 29828 raft_consensus.cc:2258] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.342845 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1171) took 1932 ms. Trace:
I20240503 14:12:33.344606 28661 rpcz_store.cc:276] 0503 14:12:31.410010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.410304 (+ 294us) service_pool.cc:225] Handling call
0503 14:12:33.338046 (+1927742us) ts_tablet_manager.cc:1120] Deleting tablet 9028a6990d0840ffa6849143f6c004db
0503 14:12:33.338057 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.342815 (+ 4758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1918051}
I20240503 14:12:33.344997 29828 ts_tablet_manager.cc:1875] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.337888 30729 raft_consensus.cc:2229] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.347374 30729 raft_consensus.cc:2258] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.347973 28661 tablet_replica.cc:330] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:33.348361 28661 raft_consensus.cc:2229] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:33.348505 30729 ts_tablet_manager.cc:1875] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.348621 28661 raft_consensus.cc:2258] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.349309 28661 ts_tablet_manager.cc:1875] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.351536 30762 raft_consensus.cc:482] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.351784 30762 raft_consensus.cc:504] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.352589 30762 leader_election.cc:288] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.346504 28604 tablet_bootstrap.cc:654] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.373684 29828 ts_tablet_manager.cc:1888] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.374286 29828 log.cc:1196] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/646f83185d5241268d4126d10561e937
I20240503 14:12:33.375250 29828 ts_tablet_manager.cc:1909] T 646f83185d5241268d4126d10561e937 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.376525 28600 tablet_bootstrap.cc:492] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:33.376765 29828 tablet_replica.cc:330] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.377095 28600 ts_tablet_manager.cc:1367] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.411s user 0.022s sys 0.031s
I20240503 14:12:33.377642 29828 raft_consensus.cc:2229] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.378247 29828 pending_rounds.cc:62] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:33.378949 29828 pending_rounds.cc:68] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597778641088512 op_type: NO_OP noop_request { }
I20240503 14:12:33.380065 29828 raft_consensus.cc:2866] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:33.381971 29828 raft_consensus.cc:2258] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.382709 28600 raft_consensus.cc:348] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.384480 29828 ts_tablet_manager.cc:1875] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.384522 28600 raft_consensus.cc:374] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.385124 28600 raft_consensus.cc:729] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.386152 28604 tablet_bootstrap.cc:492] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:33.386404 28604 ts_tablet_manager.cc:1367] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20240503 14:12:33.387501 28604 raft_consensus.cc:348] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.387847 28604 raft_consensus.cc:374] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.387951 28604 raft_consensus.cc:729] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.386948 28600 consensus_queue.cc:260] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.388348 28604 consensus_queue.cc:260] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.388468 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet f080c95ff5ca41e0b3d27382040107e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:16 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.389045 28427 leader_election.cc:302] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:33.389351 28604 ts_tablet_manager.cc:1398] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:33.392045 28600 ts_tablet_manager.cc:1398] T cc939b81d7cc472aa73b51d9dc397013 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240503 14:12:33.393406 30739 raft_consensus.cc:482] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.393568 30739 raft_consensus.cc:504] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.394330 30739 leader_election.cc:288] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:33.394405 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet d9c7b5fb9b4a483c8292f68387591034 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a9f6a750b744433b01f821506767a02) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.394351 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet 6809b59e87644316b9a450d6b670da2b with delete_type TABLET_DATA_DELETED (replaced by tablet 5d8ddcdb3c764dc59e9291725046d839) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.394697 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b468dd4102495abd4337edcfc6340a with delete_type TABLET_DATA_DELETED (replaced by tablet f783ee62adba49579a0518b240b795bc) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.395830 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218481d1ddd246cf9aa9c0e163666f4e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.395978 28600 tablet_bootstrap.cc:492] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:33.389454 30762 raft_consensus.cc:2781] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:33.395491 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218481d1ddd246cf9aa9c0e163666f4e" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:33.396454 28150 leader_election.cc:341] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.396776 30762 raft_consensus.cc:686] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
W20240503 14:12:33.397094 28150 leader_election.cc:341] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.397205 28150 leader_election.cc:302] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:33.397315 28604 tablet_bootstrap.cc:492] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:33.397231 30762 consensus_queue.cc:237] T a1ebec4622ef4950bd1c70451958c59b P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } }
I20240503 14:12:33.400385 30739 raft_consensus.cc:2726] T 218481d1ddd246cf9aa9c0e163666f4e P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:33.400897 28604 tablet_bootstrap.cc:654] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.408835 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7d348da459634561985c4ac1710c0bb2 (table default.Table_ebd8ae7785ef42cca8a6b747de4e0285 [id=7552d7d498de405fbddad0211f40e0b6]) successfully deleted
I20240503 14:12:33.409268 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 31ddd791d4e14630bfb3b4b1513f76e1 (table default.Table_cf24dd3520bd4373acfd719f2c7737ad [id=e0a3f887fca8477ba19c4ed3738b1955]) successfully deleted
W20240503 14:12:33.409643 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6de9e159d2ba47039dd6bd207a9cddeb because the tablet was not found. No further retry: Not found: Tablet not found: 6de9e159d2ba47039dd6bd207a9cddeb
I20240503 14:12:33.410151 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet d7c79ffee3954b5c870f4a8aa6379517 (table default.Table_8f505fa519514278ae7489b9ff7bbaf2 [id=f58e9b32e96948cfab2760b64eb18b68]) successfully deleted
W20240503 14:12:33.410487 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet cf810e84367f420b928a171790849445 because the tablet was not found. No further retry: Not found: Tablet not found: cf810e84367f420b928a171790849445
I20240503 14:12:33.411793 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 08cc84cf703048148abfe03b7dee12ce (table default.Table_20718ac286c9405b871c43b98ac94f8d [id=51930b6b93bd406581a479188de3423d]) successfully deleted
I20240503 14:12:33.412336 30733 raft_consensus.cc:482] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.412518 30733 raft_consensus.cc:504] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.413264 30733 leader_election.cc:288] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.417641 28661 ts_tablet_manager.cc:1888] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.418008 28661 log.cc:1196] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/989ea2f962a7492d8c32e4becec1457d
I20240503 14:12:33.418269 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7b805aee554f8fa4c198ce3409c6e6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.418532 28209 raft_consensus.cc:2454] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:33.419869 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a9d58fb30d4b60ae0a44ec217fea11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.420178 28209 raft_consensus.cc:2454] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:33.421113 28600 tablet_bootstrap.cc:654] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.423580 28661 ts_tablet_manager.cc:1909] T 989ea2f962a7492d8c32e4becec1457d P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:33.424911 28290 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.50.195:35233, user_credentials={real_user=slave}} blocked reactor thread for 167057us
I20240503 14:12:33.427505 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a9d58fb30d4b60ae0a44ec217fea11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.427816 30729 ts_tablet_manager.cc:1888] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.427955 30729 log.cc:1196] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/2b52ef7ecedf4273939406581d843980
I20240503 14:12:33.428304 30729 ts_tablet_manager.cc:1909] T 2b52ef7ecedf4273939406581d843980 P a64bc15d65d84ad5a005960a9a84df9c: Deleting consensus metadata
I20240503 14:12:33.429363 30729 tablet_replica.cc:330] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: stopping tablet replica
I20240503 14:12:33.431538 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:17 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.431821 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet b59945d011b64cba87c854e2ffde8226 with delete_type TABLET_DATA_DELETED (replaced by tablet 47965a978a904f2d876690de01f89771) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.431995 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet aac51b83abe14ab5b1fb5d627233e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.425451 28427 leader_election.cc:302] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:33.435547 28427 leader_election.cc:341] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:33.435931 28427 consensus_peers.cc:485] T ac752d8ab4fd4a218b052f388f416494 P d472d9f94d8949cf86284d32a161cebc -> Peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Couldn't send request to peer d3a6b9b7913444d29c96581af6febf33. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:33.436543 30733 raft_consensus.cc:2781] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:33.436671 30733 raft_consensus.cc:686] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:33.436964 30733 consensus_queue.cc:237] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.439980 30739 raft_consensus.cc:482] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.440202 30766 raft_consensus.cc:482] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d472d9f94d8949cf86284d32a161cebc)
I20240503 14:12:33.440205 30739 raft_consensus.cc:504] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.440338 30766 raft_consensus.cc:504] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.426219 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b86451bc448428da7485e70e0d54b89" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:33.440930 30766 leader_election.cc:288] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.440953 30739 leader_election.cc:288] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:33.442070 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d7687bbd9c41e49f2776019683a32f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
W20240503 14:12:33.442500 27856 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.27.50.194:33463, user_credentials={real_user=slave}} blocked reactor thread for 283080us
I20240503 14:12:33.442596 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d7687bbd9c41e49f2776019683a32f" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.445080 29828 ts_tablet_manager.cc:1888] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240503 14:12:33.445281 28150 leader_election.cc:341] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:33.445518 28150 leader_election.cc:341] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.445595 28150 leader_election.cc:302] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c; no voters: d3a6b9b7913444d29c96581af6febf33, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:33.445677 29828 log.cc:1196] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/7b86451bc448428da7485e70e0d54b89
I20240503 14:12:33.446090 29828 ts_tablet_manager.cc:1909] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.446318 30739 raft_consensus.cc:2726] T d5d7687bbd9c41e49f2776019683a32f P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:33.449311 29828 tablet_replica.cc:330] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.459173 29828 raft_consensus.cc:2229] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.463956 29828 raft_consensus.cc:2258] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.465176 29828 ts_tablet_manager.cc:1875] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.424059 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1172) took 1854 ms. Trace:
I20240503 14:12:33.465847 28661 rpcz_store.cc:276] 0503 14:12:31.569389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.569480 (+ 91us) service_pool.cc:225] Handling call
0503 14:12:33.347897 (+1778417us) ts_tablet_manager.cc:1120] Deleting tablet 989ea2f962a7492d8c32e4becec1457d
0503 14:12:33.347910 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.369491 (+ 21581us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:33.417478 (+ 47987us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:33.424030 (+ 6552us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1778183}
I20240503 14:12:33.466717 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1174) took 1682 ms. Trace:
I20240503 14:12:33.466928 28661 rpcz_store.cc:276] 0503 14:12:31.784684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.784738 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:33.466539 (+1681801us) ts_tablet_manager.cc:1120] Deleting tablet ad9baf131a1948d8ae5fae4c706aa726
0503 14:12:33.466555 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.466694 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681604}
I20240503 14:12:33.467120 28600 tablet_bootstrap.cc:492] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:33.467375 28600 ts_tablet_manager.cc:1367] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.003s
I20240503 14:12:33.467654 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1175) took 1682 ms. Trace:
I20240503 14:12:33.467784 28661 rpcz_store.cc:276] 0503 14:12:31.785075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.785121 (+ 46us) service_pool.cc:225] Handling call
0503 14:12:33.467524 (+1682403us) ts_tablet_manager.cc:1120] Deleting tablet ceec6992b0474706b8b3766518b0c274
0503 14:12:33.467535 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.467636 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682252}
I20240503 14:12:33.468748 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1176) took 1683 ms. Trace:
I20240503 14:12:33.468739 28600 raft_consensus.cc:348] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.469161 28600 raft_consensus.cc:374] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.469301 28600 raft_consensus.cc:729] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.469730 28600 consensus_queue.cc:260] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.472950 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f81bbdb2ee149c5a446c608034585fe" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.473172 28212 raft_consensus.cc:2454] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:33.474968 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788e4bfa1e414630939bc19da1ede3fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.475180 28212 raft_consensus.cc:2454] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
W20240503 14:12:33.478251 28017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.50.193:44355, user_credentials={real_user=slave}} blocked reactor thread for 66454.2us
I20240503 14:12:33.478606 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet a26e8389639748ae99832d7a7a9d5af3 (table default.Table_5bbe821329104389be37f7890f051041 [id=78e7487e239c4115ae6f3a3a45ca8b9b]) successfully deleted
I20240503 14:12:33.483979 28600 ts_tablet_manager.cc:1398] T 2b9a6aca21d746eb8f295ed5ea44c9d9 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:12:33.486534 28600 tablet_bootstrap.cc:492] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:33.487728 30729 raft_consensus.cc:2229] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.488188 30729 pending_rounds.cc:62] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Trying to abort 1 pending ops.
I20240503 14:12:33.488284 30729 pending_rounds.cc:68] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597779483348992 op_type: NO_OP noop_request { }
I20240503 14:12:33.488456 30729 raft_consensus.cc:2866] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:33.488584 30729 raft_consensus.cc:2258] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.492664 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 31ddd791d4e14630bfb3b4b1513f76e1 (table default.Table_cf24dd3520bd4373acfd719f2c7737ad [id=e0a3f887fca8477ba19c4ed3738b1955]) successfully deleted
W20240503 14:12:33.493016 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 01482b478dcd45598fa65d14904fe276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01482b478dcd45598fa65d14904fe276 already in progress: creating tablet
W20240503 14:12:33.493280 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 61da48859c4d4a73bb1b9d7c0ede8106 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61da48859c4d4a73bb1b9d7c0ede8106 already in progress: creating tablet
I20240503 14:12:33.493371 29828 ts_tablet_manager.cc:1888] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240503 14:12:33.493535 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet a88cf5de223f4a6abc21691fea1c4da0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88cf5de223f4a6abc21691fea1c4da0 already in progress: creating tablet
I20240503 14:12:33.493867 29828 log.cc:1196] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/9b21cd1262d94d1c994b29c44f86cb5e
W20240503 14:12:33.494046 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 530f24322a454f7bb04e818ca865a369 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 530f24322a454f7bb04e818ca865a369 already in progress: creating tablet
W20240503 14:12:33.494442 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet f783ee62adba49579a0518b240b795bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f783ee62adba49579a0518b240b795bc already in progress: creating tablet
W20240503 14:12:33.495136 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 9028a6990d0840ffa6849143f6c004db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9028a6990d0840ffa6849143f6c004db already in progress: creating tablet
W20240503 14:12:33.490550 28427 consensus_peers.cc:485] T a05f786073084bd1b8c472c20dc8bff6 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a05f786073084bd1b8c472c20dc8bff6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:33.495740 28427 leader_election.cc:302] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters: d3a6b9b7913444d29c96581af6febf33
I20240503 14:12:33.496030 29828 ts_tablet_manager.cc:1909] T 9b21cd1262d94d1c994b29c44f86cb5e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.496006 28017 catalog_manager.cc:4830] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): tablet 989ea2f962a7492d8c32e4becec1457d (table default.Table_da6c5739b62c43758e9e53182b790622 [id=6b342f8063bb4c2db3d8f63359afa1a8]) successfully deleted
W20240503 14:12:33.496230 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ad9baf131a1948d8ae5fae4c706aa726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad9baf131a1948d8ae5fae4c706aa726 already in progress: creating tablet
W20240503 14:12:33.496441 28427 leader_election.cc:341] T 996bb3a7edfc4f5a8a3a39ea6fcfe5cd P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 996bb3a7edfc4f5a8a3a39ea6fcfe5cd
I20240503 14:12:33.496886 28427 leader_election.cc:302] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:33.497496 30666 raft_consensus.cc:2781] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:33.497578 30666 raft_consensus.cc:482] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:33.497634 30666 raft_consensus.cc:3037] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.499310 29828 tablet_replica.cc:330] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.499522 29828 raft_consensus.cc:2229] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.499682 29828 raft_consensus.cc:2258] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.500375 30729 ts_tablet_manager.cc:1875] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.500172 28600 tablet_bootstrap.cc:654] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.500562 30666 raft_consensus.cc:504] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.501873 30666 leader_election.cc:288] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.502254 30666 raft_consensus.cc:2781] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:33.502449 30666 raft_consensus.cc:482] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:33.502563 30666 raft_consensus.cc:3037] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.505244 29828 ts_tablet_manager.cc:1875] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.505338 30666 raft_consensus.cc:504] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.506958 30666 leader_election.cc:288] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.507813 30558 consensus_queue.cc:1035] T 888636e9beca43edab96a8736751409e P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:33.516129 28427 leader_election.cc:302] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:33.516438 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ceec6992b0474706b8b3766518b0c274 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceec6992b0474706b8b3766518b0c274 already in progress: creating tablet
I20240503 14:12:33.517082 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet ac752d8ab4fd4a218b052f388f416494 (table default.Table_c7e5a76bf36545a5b53fc3d60512a521 [id=e6889b2bf6214cabba6669c414c22f7a]) successfully deleted
I20240503 14:12:33.517338 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 6696a237c3dd428097e2ebdf24afd63f (table default.Table_43c9760cf3b54d68b218cafe8393cda2 [id=821a0387d6f44024b05959350fd53b63]) successfully deleted
I20240503 14:12:33.517547 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 646f83185d5241268d4126d10561e937 (table default.Table_4d256c29f9b7400497caee67285a9831 [id=6c9175a58b8544798ce769460dfd9cf5]) successfully deleted
I20240503 14:12:33.516459 30716 consensus_queue.cc:1035] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:33.518177 30716 raft_consensus.cc:2781] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:33.518278 30716 raft_consensus.cc:482] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:33.518373 30716 raft_consensus.cc:3037] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.518373 29828 ts_tablet_manager.cc:1888] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 14:12:33.518484 29828 log.cc:1196] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/b59945d011b64cba87c854e2ffde8226
I20240503 14:12:33.519969 28024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 393f27164887427d8a654c51cfc04e14
I20240503 14:12:33.520061 30558 raft_consensus.cc:482] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.520223 30558 raft_consensus.cc:504] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.520896 30558 leader_election.cc:288] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.522471 28427 leader_election.cc:302] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:33.522799 28032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet f0a278bb9fe04a1a93c15da6cf38d27e
I20240503 14:12:33.522984 30558 raft_consensus.cc:2781] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:33.523051 30558 raft_consensus.cc:482] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:33.523102 30558 raft_consensus.cc:3037] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.523535 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_086b42c569ed4cea9515163af7550f8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:33.524334 28028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet 6a9f6a750b744433b01f821506767a02
I20240503 14:12:33.525035 28029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet be58d7b6fdf24bf1ad421182b660c77c
I20240503 14:12:33.525758 28029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_f010ec576f0a40cb91c71b23da613670" } new_table_name: "default.Table_a1140ed619264bf1835567f9115b7fd6" modify_external_catalogs: true
I20240503 14:12:33.526001 30558 raft_consensus.cc:504] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.526643 30558 leader_election.cc:288] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.527297 28030 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet db88ca39eba44740994dd05b526e8506
I20240503 14:12:33.527683 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7b805aee554f8fa4c198ce3409c6e6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:33.527892 28212 raft_consensus.cc:3037] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.528120 28030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_dba60f9f02f94c63a19289c80be29282"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:33.528738 28030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dba60f9f02f94c63a19289c80be29282 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.530710 28212 raft_consensus.cc:2454] T 2f7b805aee554f8fa4c198ce3409c6e6 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:33.531805 28661 rpcz_store.cc:276] 0503 14:12:31.785420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.785462 (+ 42us) service_pool.cc:225] Handling call
0503 14:12:33.468616 (+1683154us) ts_tablet_manager.cc:1120] Deleting tablet ac0813ac39364a268dd7df98d32403c6
0503 14:12:33.468626 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.468727 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682997}
I20240503 14:12:33.532701 28026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: Sending DeleteTablet for 3 replicas of tablet adc4677a01f14f7c87ca00b29c6238c6
I20240503 14:12:33.534394 28028 catalog_manager.cc:6554] ReplaceTablet: tablet 6a9f6a750b744433b01f821506767a02 deleted and replaced by tablet 0930a317acc942fdb50d26b08ff5e269
I20240503 14:12:33.537772 28026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_61e0a4e7d7544862b3028e7b3eec5bed" } new_table_name: "default.Table_aff9b454a0e84ac5bcd9848741ffbfb1" modify_external_catalogs: true
I20240503 14:12:33.538012 28032 catalog_manager.cc:5576] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355) does not have the latest schema for tablet 0f223cbbcaa5402a888b084df0fe1c27 (table default.Table_ec97037ea5a849bfb48a72d6771fbf1f [id=e6871aca8d244f00be81cf65dd32a18e]). Expected version 2 got 0
W20240503 14:12:33.540227 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_086b42c569ed4cea9515163af7550f8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.541986 28028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_3c99c8cdca9c4ee0a96e4bbf77e68eb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:33.542690 28028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c99c8cdca9c4ee0a96e4bbf77e68eb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.544857 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a9d58fb30d4b60ae0a44ec217fea11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:33.545101 28212 raft_consensus.cc:3037] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.548306 28212 raft_consensus.cc:2454] T 74a9d58fb30d4b60ae0a44ec217fea11 P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:33.550509 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1177) took 1764 ms. Trace:
I20240503 14:12:33.555588 30729 ts_tablet_manager.cc:1888] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.556576 30729 log.cc:1196] T f8fe462c9ee54a27b9714f6457036d76 P a64bc15d65d84ad5a005960a9a84df9c: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-0/wal/wals/f8fe462c9ee54a27b9714f6457036d76
I20240503 14:12:33.561043 28017 catalog_manager.cc:4830] TS a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): tablet 2b52ef7ecedf4273939406581d843980 (table default.Table_07fdeadf61a045d8aeb377c2a3c4c1af [id=be2dbf4734c74e93ad430d203979e633]) successfully deleted
I20240503 14:12:33.553500 28604 tablet_bootstrap.cc:492] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:33.561704 28604 ts_tablet_manager.cc:1367] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.164s user 0.012s sys 0.010s
I20240503 14:12:33.562767 28604 raft_consensus.cc:348] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.563028 28604 raft_consensus.cc:374] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.563091 28604 raft_consensus.cc:729] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.563316 28604 consensus_queue.cc:260] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.564133 28604 ts_tablet_manager.cc:1398] T 0a6a58db50f849a8be729c10dbe03e76 P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:33.556063 30778 tablet.cc:1722] T 0f223cbbcaa5402a888b084df0fe1c27 P d472d9f94d8949cf86284d32a161cebc: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:33.564647 30716 raft_consensus.cc:504] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.565477 30716 leader_election.cc:288] T 3f81bbdb2ee149c5a446c608034585fe P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Requested vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.573166 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:33.561561 28661 rpcz_store.cc:276] 0503 14:12:31.785787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:31.785826 (+ 39us) service_pool.cc:225] Handling call
0503 14:12:33.532154 (+1746328us) ts_tablet_manager.cc:1120] Deleting tablet 328bb280f36d4c79b2795c10df2b836b
0503 14:12:33.532168 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.550471 (+ 18303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746195}
I20240503 14:12:33.574599 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1193) took 1092 ms. Trace:
I20240503 14:12:33.574707 28661 rpcz_store.cc:276] 0503 14:12:32.481821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:32.483894 (+ 2073us) service_pool.cc:225] Handling call
0503 14:12:33.574465 (+1090571us) ts_tablet_manager.cc:1120] Deleting tablet e44f92cd95164ed09ec0865bb87a6123
0503 14:12:33.574474 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.574578 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090242}
I20240503 14:12:33.575004 28661 tablet_replica.cc:330] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: stopping tablet replica
I20240503 14:12:33.575203 28661 raft_consensus.cc:2229] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:33.578636 28604 tablet_bootstrap.cc:492] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:33.581555 28604 tablet_bootstrap.cc:654] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.587968 28466 tablet_service.cc:1514] Processing DeleteTablet for tablet 393f27164887427d8a654c51cfc04e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.518769 29828 ts_tablet_manager.cc:1909] T b59945d011b64cba87c854e2ffde8226 P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.520833 28614 tablet_bootstrap.cc:492] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:33.544814 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788e4bfa1e414630939bc19da1ede3fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.589085 28614 ts_tablet_manager.cc:1367] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.555s user 0.034s sys 0.049s
I20240503 14:12:33.588191 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a278bb9fe04a1a93c15da6cf38d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.588304 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet be58d7b6fdf24bf1ad421182b660c77c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.568495 28345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7b805aee554f8fa4c198ce3409c6e6" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:33.588440 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet adc4677a01f14f7c87ca00b29c6238c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.588541 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9f6a750b744433b01f821506767a02 with delete_type TABLET_DATA_DELETED (replaced by tablet 0930a317acc942fdb50d26b08ff5e269) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.569000 28355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788e4bfa1e414630939bc19da1ede3fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:33.589916 28614 raft_consensus.cc:348] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.590023 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet db88ca39eba44740994dd05b526e8506 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.590092 28614 raft_consensus.cc:374] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.590142 28614 raft_consensus.cc:729] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.571113 28346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a9d58fb30d4b60ae0a44ec217fea11" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:33.571197 28354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45df464f6954a56bec8b107b350320c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.590344 28614 consensus_queue.cc:260] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.591056 28614 ts_tablet_manager.cc:1398] T d6f5103b31ed45b480252dcf94b72f9c P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:33.591392 28614 tablet_bootstrap.cc:492] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:33.593600 28614 tablet_bootstrap.cc:654] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:33.595417 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet ac0813ac39364a268dd7df98d32403c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac0813ac39364a268dd7df98d32403c6 already in progress: creating tablet
I20240503 14:12:33.595846 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 7b86451bc448428da7485e70e0d54b89 (table default.Table_64de235d5f5043b18820b85f48405374 [id=67ca211145f346e39e712d82700f99a0]) successfully deleted
W20240503 14:12:33.596122 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet a05f786073084bd1b8c472c20dc8bff6 because the tablet was not found. No further retry: Not found: Tablet not found: a05f786073084bd1b8c472c20dc8bff6
I20240503 14:12:33.596305 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet 9b21cd1262d94d1c994b29c44f86cb5e (table default.Table_40cc648a463747e68942c60a9d687deb [id=2bd0e4026ed5471aa1a457ebea173f93]) successfully deleted
W20240503 14:12:33.596500 28017 catalog_manager.cc:4797] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 5cdc868ddf0a490292eae78ebc8c8ec6 because the tablet was not found. No further retry: Not found: Tablet not found: 5cdc868ddf0a490292eae78ebc8c8ec6
I20240503 14:12:33.596774 28661 raft_consensus.cc:2258] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.597681 29828 tablet_replica.cc:330] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.597990 29828 raft_consensus.cc:2229] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:33.598277 29828 pending_rounds.cc:62] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Trying to abort 1 pending ops.
I20240503 14:12:33.598330 29828 pending_rounds.cc:68] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023597786273189888 op_type: NO_OP noop_request { }
I20240503 14:12:33.598460 29828 raft_consensus.cc:2866] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:12:33.598548 29828 raft_consensus.cc:2258] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:33.601068 28031 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ac7b6df20b14d1ead036cdc42174d3a consensus state has no leader
W20240503 14:12:33.601856 28029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b828eb16891434f9031b9e4d29b18d8 has no consensus state
W20240503 14:12:33.604691 28026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6cf8f068f8424317bbfce6ff10bba61f consensus state has no leader
I20240503 14:12:33.604739 28661 ts_tablet_manager.cc:1875] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.604820 29828 ts_tablet_manager.cc:1875] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.602437 28025 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c0690a516c89446d85c88cdf64fa4238: default.Table_8eb9b1e15db5432e8c3398239f7584b1 [id=e3a00fad0fa8492faf33675df3d18f9c] alter complete (version 2)
I20240503 14:12:33.607210 28025 master_service.cc:946] ReplaceTablet: received request to replace tablet 6615052ddbde487e840b6fd77a140e58 from {username='slave'} at 127.0.0.1:33344
W20240503 14:12:33.610600 28427 leader_election.cc:341] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.612239 30446 consensus_queue.cc:1035] T 167a926fa61747e2b4fcb52ba4eeea5b P d472d9f94d8949cf86284d32a161cebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:33.574211 28290 leader_election.cc:341] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Not found: Tablet not found: 7b86451bc448428da7485e70e0d54b89
I20240503 14:12:33.614415 28290 leader_election.cc:302] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
W20240503 14:12:33.614264 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet 328bb280f36d4c79b2795c10df2b836b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 328bb280f36d4c79b2795c10df2b836b already in progress: creating tablet
W20240503 14:12:33.614621 28290 raft_consensus.cc:2670] T 7b86451bc448428da7485e70e0d54b89 P d3a6b9b7913444d29c96581af6febf33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240503 14:12:33.543565 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f81bbdb2ee149c5a446c608034585fe" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.615630 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 393f27164887427d8a654c51cfc04e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.615885 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a278bb9fe04a1a93c15da6cf38d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.616111 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet be58d7b6fdf24bf1ad421182b660c77c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.616302 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet db88ca39eba44740994dd05b526e8506 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.616494 28333 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9f6a750b744433b01f821506767a02 with delete_type TABLET_DATA_DELETED (replaced by tablet 0930a317acc942fdb50d26b08ff5e269) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.616670 28337 tablet_service.cc:1514] Processing DeleteTablet for tablet e220336186f648419b186feada6f0124 with delete_type TABLET_DATA_DELETED (replaced by tablet a83532bf2852473a9c51ce7cc78b5d34) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.616797 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet adc4677a01f14f7c87ca00b29c6238c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.620676 30716 raft_consensus.cc:482] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.620918 30716 raft_consensus.cc:504] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.621289 28604 tablet_bootstrap.cc:492] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:33.621441 28604 ts_tablet_manager.cc:1367] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240503 14:12:33.621856 30716 leader_election.cc:288] T ea261d8d5d174bee804aa2e171f35008 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.622401 28604 raft_consensus.cc:348] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.622618 28604 raft_consensus.cc:374] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.622682 28604 raft_consensus.cc:729] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.622896 28604 consensus_queue.cc:260] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.624231 28318 tablet_service.cc:1514] Processing DeleteTablet for tablet 996bb3a7edfc4f5a8a3a39ea6fcfe5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:58028
I20240503 14:12:33.624656 28604 ts_tablet_manager.cc:1398] T 08cc84cf703048148abfe03b7dee12ce P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:33.625018 28604 tablet_bootstrap.cc:492] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
I20240503 14:12:33.625574 28614 tablet_bootstrap.cc:492] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33: No bootstrap required, opened a new log
I20240503 14:12:33.626525 28614 ts_tablet_manager.cc:1367] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20240503 14:12:33.626926 28427 leader_election.cc:302] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
W20240503 14:12:33.617786 28017 catalog_manager.cc:4810] TS d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233): delete failed for tablet e44f92cd95164ed09ec0865bb87a6123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e44f92cd95164ed09ec0865bb87a6123 already in progress: creating tablet
I20240503 14:12:33.628382 30446 raft_consensus.cc:2781] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:33.628522 30446 raft_consensus.cc:686] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:33.628768 30446 consensus_queue.cc:237] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.630798 30666 raft_consensus.cc:482] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.631008 30666 raft_consensus.cc:504] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.631732 30666 leader_election.cc:288] T a2369dd6d16e4b9681108f2c8574f658 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.627233 28604 tablet_bootstrap.cc:654] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.633749 28427 leader_election.cc:302] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc; no voters:
I20240503 14:12:33.636245 30666 raft_consensus.cc:2781] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:33.635346 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet b59945d011b64cba87c854e2ffde8226 (table default.Table_3949c7d2081d40b78d82c0b7a42e5dfa [id=7044f389a80e486f81a1debfbd2ae68a]) successfully deleted
I20240503 14:12:33.636379 30666 raft_consensus.cc:686] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [term 1 LEADER]: Becoming Leader. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Running, Role: LEADER
I20240503 14:12:33.636619 30666 consensus_queue.cc:237] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.638685 28614 raft_consensus.cc:348] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:33.640380 28427 consensus_peers.cc:485] T f8fe462c9ee54a27b9714f6457036d76 P d472d9f94d8949cf86284d32a161cebc -> Peer a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355): Couldn't send request to peer a64bc15d65d84ad5a005960a9a84df9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:33.643776 28661 ts_tablet_manager.cc:1888] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.639297 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f81bbdb2ee149c5a446c608034585fe" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33"
I20240503 14:12:33.647066 29828 ts_tablet_manager.cc:1888] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 14:12:33.647223 29828 log.cc:1196] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-1/wal/wals/aac51b83abe14ab5b1fb5d627233e58e
I20240503 14:12:33.647248 28614 raft_consensus.cc:374] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.648057 28614 raft_consensus.cc:729] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a6b9b7913444d29c96581af6febf33, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.559788 30739 raft_consensus.cc:482] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.561110 28600 tablet_bootstrap.cc:492] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:33.648445 30765 raft_consensus.cc:482] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.650103 28600 ts_tablet_manager.cc:1367] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.164s user 0.009s sys 0.000s
I20240503 14:12:33.650034 30739 raft_consensus.cc:504] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.647605 29828 ts_tablet_manager.cc:1909] T aac51b83abe14ab5b1fb5d627233e58e P d3a6b9b7913444d29c96581af6febf33: Deleting consensus metadata
I20240503 14:12:33.650151 30765 raft_consensus.cc:504] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.650672 28614 consensus_queue.cc:260] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.650804 30739 leader_election.cc:288] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463), d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233)
I20240503 14:12:33.650852 28600 raft_consensus.cc:348] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.651049 28600 raft_consensus.cc:374] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.651113 28600 raft_consensus.cc:729] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.562381 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 393f27164887427d8a654c51cfc04e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:32 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.562484 28180 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a278bb9fe04a1a93c15da6cf38d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.562904 28192 tablet_service.cc:1514] Processing DeleteTablet for tablet be58d7b6fdf24bf1ad421182b660c77c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.563063 28188 tablet_service.cc:1514] Processing DeleteTablet for tablet db88ca39eba44740994dd05b526e8506 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.563810 28186 tablet_service.cc:1514] Processing DeleteTablet for tablet adc4677a01f14f7c87ca00b29c6238c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:33 UTC) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.563987 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9f6a750b744433b01f821506767a02 with delete_type TABLET_DATA_DELETED (replaced by tablet 0930a317acc942fdb50d26b08ff5e269) from {username='slave'} at 127.0.0.1:58172
I20240503 14:12:33.585306 28206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45df464f6954a56bec8b107b350320c" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.652608 28206 raft_consensus.cc:2454] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:33.585556 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888636e9beca43edab96a8736751409e" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.586002 30759 tablet.cc:1722] T 0f223cbbcaa5402a888b084df0fe1c27 P a64bc15d65d84ad5a005960a9a84df9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240503 14:12:33.586658 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788e4bfa1e414630939bc19da1ede3fb" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:33.653551 28207 raft_consensus.cc:3037] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.650840 30765 leader_election.cc:288] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), a64bc15d65d84ad5a005960a9a84df9c (127.27.50.193:44355)
I20240503 14:12:33.655210 28614 ts_tablet_manager.cc:1398] T f8fe462c9ee54a27b9714f6457036d76 P d3a6b9b7913444d29c96581af6febf33: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240503 14:12:33.655542 28614 tablet_bootstrap.cc:492] T c00dbc5a90674873b61b377e3e8d92fb P d3a6b9b7913444d29c96581af6febf33: Bootstrap starting.
I20240503 14:12:33.589264 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f81bbdb2ee149c5a446c608034585fe" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c"
I20240503 14:12:33.626212 30788 raft_consensus.cc:482] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:33.651324 28600 consensus_queue.cc:260] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.656248 28215 raft_consensus.cc:3037] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:33.656616 28600 ts_tablet_manager.cc:1398] T cf221f17ddfd461f83f7f916d7833011 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:33.656915 28600 tablet_bootstrap.cc:492] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:33.658385 28600 tablet_bootstrap.cc:654] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:33.658681 29828 tablet_replica.cc:330] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33: stopping tablet replica
I20240503 14:12:33.659375 28215 raft_consensus.cc:2454] T 3f81bbdb2ee149c5a446c608034585fe P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
I20240503 14:12:33.660104 29828 raft_consensus.cc:2229] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:33.660300 29828 raft_consensus.cc:2258] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:33.661974 30788 raft_consensus.cc:504] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.662153 29828 ts_tablet_manager.cc:1875] T e220336186f648419b186feada6f0124 P d3a6b9b7913444d29c96581af6febf33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:12:33.664615 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45df464f6954a56bec8b107b350320c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.664703 30788 leader_election.cc:288] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d472d9f94d8949cf86284d32a161cebc (127.27.50.195:35233), d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463)
I20240503 14:12:33.666472 28207 raft_consensus.cc:2454] T 788e4bfa1e414630939bc19da1ede3fb P a64bc15d65d84ad5a005960a9a84df9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 1.
W20240503 14:12:33.666936 28150 leader_election.cc:341] T e45df464f6954a56bec8b107b350320c P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.667212 28353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6809b59e87644316b9a450d6b670da2b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
W20240503 14:12:33.667912 28150 leader_election.cc:341] T 6809b59e87644316b9a450d6b670da2b P a64bc15d65d84ad5a005960a9a84df9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.669116 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea261d8d5d174bee804aa2e171f35008" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.669394 28207 raft_consensus.cc:2454] T ea261d8d5d174bee804aa2e171f35008 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:33.669762 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2369dd6d16e4b9681108f2c8574f658" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64bc15d65d84ad5a005960a9a84df9c" is_pre_election: true
I20240503 14:12:33.669992 28215 raft_consensus.cc:2454] T a2369dd6d16e4b9681108f2c8574f658 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d472d9f94d8949cf86284d32a161cebc in term 0.
I20240503 14:12:33.672116 28017 catalog_manager.cc:4830] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): tablet aac51b83abe14ab5b1fb5d627233e58e (table default.Table_618386316f9b46f9a7770c3f998769f5 [id=ffcb298fb844406881d9a8812e24e896]) successfully deleted
W20240503 14:12:33.672447 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 393f27164887427d8a654c51cfc04e14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 393f27164887427d8a654c51cfc04e14 already in progress: creating tablet
W20240503 14:12:33.672717 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet f0a278bb9fe04a1a93c15da6cf38d27e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0a278bb9fe04a1a93c15da6cf38d27e already in progress: creating tablet
W20240503 14:12:33.672919 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet be58d7b6fdf24bf1ad421182b660c77c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be58d7b6fdf24bf1ad421182b660c77c already in progress: creating tablet
W20240503 14:12:33.673094 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet db88ca39eba44740994dd05b526e8506 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db88ca39eba44740994dd05b526e8506 already in progress: creating tablet
W20240503 14:12:33.673281 28017 catalog_manager.cc:4810] TS d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): delete failed for tablet 6a9f6a750b744433b01f821506767a02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a9f6a750b744433b01f821506767a02 already in progress: creating tablet
I20240503 14:12:33.676438 28600 tablet_bootstrap.cc:492] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c: No bootstrap required, opened a new log
I20240503 14:12:33.676657 28600 ts_tablet_manager.cc:1367] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.003s
I20240503 14:12:33.677554 28600 raft_consensus.cc:348] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.677809 28600 raft_consensus.cc:374] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.677861 28600 raft_consensus.cc:729] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bc15d65d84ad5a005960a9a84df9c, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.678119 28600 consensus_queue.cc:260] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
W20240503 14:12:33.678467 28020 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 6cf8f068f8424317bbfce6ff10bba61f (table default.Table_aff9b454a0e84ac5bcd9848741ffbfb1 [id=770381506d7c4730849f49d87eb5e05a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6cf8f068f8424317bbfce6ff10bba61f consensus state has no leader
I20240503 14:12:33.678761 28600 ts_tablet_manager.cc:1398] T e8f9f10348014293a421ed11a50f89e7 P a64bc15d65d84ad5a005960a9a84df9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:33.679196 28600 tablet_bootstrap.cc:492] T f0287bc315644f0aaa976a56174576d9 P a64bc15d65d84ad5a005960a9a84df9c: Bootstrap starting.
I20240503 14:12:33.684468 28027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1a697aa9641d4416aa937df900982b45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:33.685197 28027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a697aa9641d4416aa937df900982b45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.686635 28024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_a5de24e133584200af645a728b1346e1" } new_table_name: "default.Table_34cdd637c31d48b184b17accd46b2034" modify_external_catalogs: true
I20240503 14:12:33.687798 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea261d8d5d174bee804aa2e171f35008" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.659804 28604 tablet_bootstrap.cc:492] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc: No bootstrap required, opened a new log
I20240503 14:12:33.688397 28661 log.cc:1196] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: Deleting WAL directory at /tmp/dist-test-taskwmRrLf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714745514850393-27851-0/minicluster-data/ts-2/wal/wals/7b86451bc448428da7485e70e0d54b89
I20240503 14:12:33.688462 28604 ts_tablet_manager.cc:1367] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.003s
I20240503 14:12:33.688277 28033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_1834650a0b8e429fabbbd93725336cd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:33.688912 28661 ts_tablet_manager.cc:1909] T 7b86451bc448428da7485e70e0d54b89 P d472d9f94d8949cf86284d32a161cebc: Deleting consensus metadata
W20240503 14:12:33.688920 28033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1834650a0b8e429fabbbd93725336cd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.689749 28661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51314 (request call id 1192) took 1208 ms. Trace:
I20240503 14:12:33.689826 28604 raft_consensus.cc:348] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.689846 28661 rpcz_store.cc:276] 0503 14:12:32.481666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:32.486911 (+ 5245us) service_pool.cc:225] Handling call
0503 14:12:33.574957 (+1088046us) ts_tablet_manager.cc:1120] Deleting tablet 7b86451bc448428da7485e70e0d54b89
0503 14:12:33.574966 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 14:12:33.608041 (+ 33075us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:33.643638 (+ 35597us) tablet_metadata.cc:641] Metadata flushed
0503 14:12:33.689723 (+ 46085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087718}
I20240503 14:12:33.690001 28604 raft_consensus.cc:374] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:33.690047 28604 raft_consensus.cc:729] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d472d9f94d8949cf86284d32a161cebc, State: Initialized, Role: FOLLOWER
I20240503 14:12:33.690380 28604 consensus_queue.cc:260] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a6b9b7913444d29c96581af6febf33" member_type: VOTER last_known_addr { host: "127.27.50.194" port: 33463 } } peers { permanent_uuid: "d472d9f94d8949cf86284d32a161cebc" member_type: VOTER last_known_addr { host: "127.27.50.195" port: 35233 } } peers { permanent_uuid: "a64bc15d65d84ad5a005960a9a84df9c" member_type: VOTER last_known_addr { host: "127.27.50.193" port: 44355 } }
I20240503 14:12:33.690474 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2369dd6d16e4b9681108f2c8574f658" candidate_uuid: "d472d9f94d8949cf86284d32a161cebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a6b9b7913444d29c96581af6febf33" is_pre_election: true
I20240503 14:12:33.663763 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ddd791d4e14630bfb3b4b1513f76e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.665727 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45df464f6954a56bec8b107b350320c" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:33.692456 28493 raft_consensus.cc:2454] T e45df464f6954a56bec8b107b350320c P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64bc15d65d84ad5a005960a9a84df9c in term 0.
I20240503 14:12:33.666466 28494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6809b59e87644316b9a450d6b670da2b" candidate_uuid: "a64bc15d65d84ad5a005960a9a84df9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:33.672148 28488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e220336186f648419b186feada6f0124" candidate_uuid: "d3a6b9b7913444d29c96581af6febf33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d472d9f94d8949cf86284d32a161cebc" is_pre_election: true
I20240503 14:12:33.692927 28290 leader_election.cc:302] T 888636e9beca43edab96a8736751409e P d3a6b9b7913444d29c96581af6febf33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a6b9b7913444d29c96581af6febf33; no voters: a64bc15d65d84ad5a005960a9a84df9c, d472d9f94d8949cf86284d32a161cebc
I20240503 14:12:33.692780 28029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33344:
name: "default.Table_06e80f6d67184b60851e01d008f1930c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:33.693043 28488 raft_consensus.cc:2454] T e220336186f648419b186feada6f0124 P d472d9f94d8949cf86284d32a161cebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a6b9b7913444d29c96581af6febf33 in term 0.
I20240503 14:12:33.693190 28614 tablet_bootstrap.cc:654] T c00dbc5a90674873b61b377e3e8d92fb P d3a6b9b7913444d29c96581af6febf33: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:33.693475 28029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06e80f6d67184b60851e01d008f1930c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:33.673620 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet 989ea2f962a7492d8c32e4becec1457d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:31 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.673836 28459 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a9d58fb30d4b60ae0a44ec217fea11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:15 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.673966 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f5103b31ed45b480252dcf94b72f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:12:10 UTC) from {username='slave'} at 127.0.0.1:51314
I20240503 14:12:33.694118 28604 ts_tablet_manager.cc:1398] T 5ac7b6df20b14d1ead036cdc42174d3a P d472d9f94d8949cf86284d32a161cebc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20240503 14:12:33.686115 28427 leader_election.cc:341] T 2f7b805aee554f8fa4c198ce3409c6e6 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.694454 28604 tablet_bootstrap.cc:492] T 2b52ef7ecedf4273939406581d843980 P d472d9f94d8949cf86284d32a161cebc: Bootstrap starting.
W20240503 14:12:33.694737 28427 leader_election.cc:341] T 788e4bfa1e414630939bc19da1ede3fb P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:12:33.694909 28427 leader_election.cc:341] T 74a9d58fb30d4b60ae0a44ec217fea11 P d472d9f94d8949cf86284d32a161cebc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3a6b9b7913444d29c96581af6febf33 (127.27.50.194:33463): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:33.694975 28032 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33344:
table { table_name: "default.Table_cca08ca8fa2f4f87a3793867e49f7b6b" } new_table_name: "default.Table_8c7c6706f21947288f5e68f3cd38da7e"